Tue Apr 30 23:37:53 UTC 2024 I: starting to build toil/unstable/armhf on jenkins on '2024-04-30 23:37' Tue Apr 30 23:37:53 UTC 2024 I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/armhf_11/2788/console.log Tue Apr 30 23:37:53 UTC 2024 I: Downloading source for unstable/toil=6.1.0-4 --2024-04-30 23:37:53-- http://deb.debian.org/debian/pool/main/t/toil/toil_6.1.0-4.dsc Connecting to 46.16.76.132:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 2774 (2.7K) [text/prs.lines.tag] Saving to: ‘toil_6.1.0-4.dsc’ 0K .. 100% 266M=0s 2024-04-30 23:37:53 (266 MB/s) - ‘toil_6.1.0-4.dsc’ saved [2774/2774] Tue Apr 30 23:37:53 UTC 2024 I: toil_6.1.0-4.dsc -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Format: 3.0 (quilt) Source: toil Binary: toil Architecture: all Version: 6.1.0-4 Maintainer: Debian Med Packaging Team Uploaders: Steffen Moeller , Michael R. Crusoe Homepage: https://github.com/BD2KGenomics/toil Standards-Version: 4.6.2 Vcs-Browser: https://salsa.debian.org/med-team/toil Vcs-Git: https://salsa.debian.org/med-team/toil.git Testsuite: autopkgtest Testsuite-Triggers: docker.io, python3-all, python3-boto3, python3-mock, python3-pytest, python3-pytest-xdist, python3-stubserver, python3-wdlparse, python3-yaml, quilt, rsync Build-Depends: debhelper-compat (= 13), dh-sequence-python3, python3-all:any, python3-setuptools, python3-configargparse, python3-docker, python3-dill, cwltool (>= 3.1.20240112164112), python3-schema-salad (>= 8.4.20230128170514), python3-enlighten, python3-wdlparse, python3-pytest , python3-pytest-xdist , python3-boto3 (>= 1.17) , python3-stubserver , rsync , python3-yaml , docker.io , python3-urllib3, python3-sphinx , python3-sphinx-autoapi , python3-sphinx-autodoc-typehints , python3-sphinxcontrib.autoprogram , python3-pubsub , python3-doc , python3-tz Package-List: toil deb net optional arch=all Checksums-Sha1: dd6f8e866e9d8d5a6430a0e75ce4b3cd830a9277 7705160 toil_6.1.0.orig.tar.gz 528de5396847af7ec69ccdd1758bd0bdca58d724 13528 toil_6.1.0-4.debian.tar.xz Checksums-Sha256: 6c7d13aac5e55fb2d515c50b8d49589c76e52132c8a1f65c3de0c1c9a518a5d2 7705160 toil_6.1.0.orig.tar.gz 7d7bd25e8010320130f2ccd07d1e62f5913d83eae7596433cab4d4c8b336231a 13528 toil_6.1.0-4.debian.tar.xz Files: a369139d8c42282a7f245846991f23a5 7705160 toil_6.1.0.orig.tar.gz 17761c6230ef3db4bb8cee80e8c34fae 13528 toil_6.1.0-4.debian.tar.xz -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEck1gkzcRPHEFUNdHPCZ2P2xn5uIFAmYvspEACgkQPCZ2P2xn 5uJb+g/+I9pF3j+5iZE4gfTUUbOB7Nj2uMJUkQkfVNIg4G4hOoJiQNYuq36GgMXZ p2T4W7OQPBbaWPs38Jv5iWeri9Y17UTLuoC8HYKOm1O06eD70Bysa5QwYHJTs9k9 Zn2IJGhAhSHIouDz88TSqeBXlI3Vxa2x+4ZydUomNlqgY0idgK7imutaOo002Z6o 0qTvrPDTCDZ582EZH4Nf7zlUlh4mw4tT01sEh7/krTpeiOUrrtdLitNm87GTIhYc 7H8DeOKyLOBzMMgJ7KmFATUnZXPszx6E32CRlzBkdxoP0kQ8s4QcAGmfhEKSk6Ka 3KnnXS0yOWKucJFPK/KlmecpVYRWeVUfJBzr91SSlb5Pyghq3EZClYXUM7i7YCoI 3WShY4iC5yI0x2HI1QB/whS4ZkSSUMe3Ul88WcusDNLCdkRufJcFKdMqjVisdMG6 IgZuJXgSMxZeDoy22EDDEnOOFXHNuFv9lekodYSiUWJTuvDOGBfa9flscLqheI11 1BJsrhxNEuOxjfY4pwRoYo6shW3w/6p4nbKjAUJs5YtracsYQ0BNc+nRWUNdDdwi Hi1QeMTBxaC0v+dReOUBE/fEJpUA2OAdYWXwfAfsxN7wQslgIW/1iwqEgXW9h1ya jDmw4iUTUQMvvr66clMRS9HAv7R71HOm2IDxo/kUhHxtkFWBiP4= =NTHN -----END PGP SIGNATURE----- Tue Apr 30 23:37:53 UTC 2024 I: Checking whether the package is not for us Tue Apr 30 23:37:53 UTC 2024 I: Starting 1st build on remote node wbq0-armhf-rb.debian.net. Tue Apr 30 23:37:53 UTC 2024 I: Preparing to do remote build '1' on wbq0-armhf-rb.debian.net. Wed May 1 00:02:12 UTC 2024 I: Deleting $TMPDIR on wbq0-armhf-rb.debian.net. I: pbuilder: network access will be disabled during build I: Current time: Tue Apr 30 11:38:08 -12 2024 I: pbuilder-time-stamp: 1714520288 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/unstable-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [toil_6.1.0-4.dsc] I: copying [./toil_6.1.0.orig.tar.gz] I: copying [./toil_6.1.0-4.debian.tar.xz] I: Extracting source gpgv: Signature made Mon Apr 29 14:45:37 2024 gpgv: using RSA key 724D609337113C710550D7473C26763F6C67E6E2 gpgv: Can't check signature: No public key dpkg-source: warning: cannot verify inline signature for ./toil_6.1.0-4.dsc: no acceptable signature found dpkg-source: info: extracting toil in toil-6.1.0 dpkg-source: info: unpacking toil_6.1.0.orig.tar.gz dpkg-source: info: unpacking toil_6.1.0-4.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying soften-configargparser-deps dpkg-source: info: applying intersphinx dpkg-source: info: applying setting_version.patch dpkg-source: info: applying no_galaxy_lib dpkg-source: info: applying debianize_docs dpkg-source: info: applying soften-mesos-deps dpkg-source: info: applying atomic_copy_as_alternative.patch dpkg-source: info: applying soften-cwltool-dep.patch dpkg-source: info: applying accept_debian_packaged_docker_version.patch I: Not using root during the build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/21397/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='armhf' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=3 ' DISTRIBUTION='unstable' HOME='/root' HOST_ARCH='armhf' IFS=' ' INVOCATION_ID='ae332d827aaf43cf9980271d71ffb049' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='21397' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.NbFkoFaV/pbuilderrc_v7Mx --distribution unstable --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/unstable-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.NbFkoFaV/b1 --logfile b1/build.log toil_6.1.0-4.dsc' SUDO_GID='113' SUDO_UID='107' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' http_proxy='http://10.0.0.15:3142/' I: uname -a Linux wbq0 6.1.0-20-armmp #1 SMP Debian 6.1.85-1 (2024-04-11) armv7l GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 Apr 25 07:46 /bin -> usr/bin I: user script /srv/workspace/pbuilder/21397/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: armhf Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper-compat (= 13), dh-sequence-python3, python3-all:any, python3-setuptools, python3-configargparse, python3-docker, python3-dill, cwltool (>= 3.1.20240112164112), python3-schema-salad (>= 8.4.20230128170514), python3-enlighten, python3-wdlparse, python3-pytest, python3-pytest-xdist, python3-boto3 (>= 1.17), python3-stubserver, rsync, python3-yaml, docker.io, python3-urllib3, python3-sphinx, python3-sphinx-autoapi, python3-sphinx-autodoc-typehints, python3-sphinxcontrib.autoprogram, python3-pubsub, python3-doc, python3-tz dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19440 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however: Package debhelper-compat is not installed. pbuilder-satisfydepends-dummy depends on dh-sequence-python3; however: Package dh-sequence-python3 is not installed. pbuilder-satisfydepends-dummy depends on python3-all:any. pbuilder-satisfydepends-dummy depends on python3-setuptools; however: Package python3-setuptools is not installed. pbuilder-satisfydepends-dummy depends on python3-configargparse; however: Package python3-configargparse is not installed. pbuilder-satisfydepends-dummy depends on python3-docker; however: Package python3-docker is not installed. pbuilder-satisfydepends-dummy depends on python3-dill; however: Package python3-dill is not installed. pbuilder-satisfydepends-dummy depends on cwltool (>= 3.1.20240112164112); however: Package cwltool is not installed. pbuilder-satisfydepends-dummy depends on python3-schema-salad (>= 8.4.20230128170514); however: Package python3-schema-salad is not installed. pbuilder-satisfydepends-dummy depends on python3-enlighten; however: Package python3-enlighten is not installed. pbuilder-satisfydepends-dummy depends on python3-wdlparse; however: Package python3-wdlparse is not installed. pbuilder-satisfydepends-dummy depends on python3-pytest; however: Package python3-pytest is not installed. pbuilder-satisfydepends-dummy depends on python3-pytest-xdist; however: Package python3-pytest-xdist is not installed. pbuilder-satisfydepends-dummy depends on python3-boto3 (>= 1.17); however: Package python3-boto3 is not installed. pbuilder-satisfydepends-dummy depends on python3-stubserver; however: Package python3-stubserver is not installed. pbuilder-satisfydepends-dummy depends on rsync; however: Package rsync is not installed. pbuilder-satisfydepends-dummy depends on python3-yaml; however: Package python3-yaml is not installed. pbuilder-satisfydepends-dummy depends on docker.io; however: Package docker.io is not installed. pbuilder-satisfydepends-dummy depends on python3-urllib3; however: Package python3-urllib3 is not installed. pbuilder-satisfydepends-dummy depends on python3-sphinx; however: Package python3-sphinx is not installed. pbuilder-satisfydepends-dummy depends on python3-sphinx-autoapi; however: Package python3-sphinx-autoapi is not installed. pbuilder-satisfydepends-dummy depends on python3-sphinx-autodoc-typehints; however: Package python3-sphinx-autodoc-typehints is not installed. pbuilder-satisfydepends-dummy depends on python3-sphinxcontrib.autoprogram; however: Package python3-sphinxcontrib.autoprogram is not installed. pbuilder-satisfydepends-dummy depends on python3-pubsub; however: Package python3-pubsub is not installed. pbuilder-satisfydepends-dummy depends on python3-doc; however: Package python3-doc is not installed. pbuilder-satisfydepends-dummy depends on python3-tz; however: Package python3-tz is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} ca-certificates{a} containerd{a} cwl-upgrader{a} cwltool{a} debhelper{a} dh-autoreconf{a} dh-python{a} dh-strip-nondeterminism{a} dmsetup{a} docker.io{a} docutils-common{a} dwz{a} file{a} fontconfig{a} fontconfig-config{a} fonts-dejavu-core{a} fonts-dejavu-mono{a} fonts-glyphicons-halflings{a} gettext{a} gettext-base{a} graphviz{a} groff-base{a} intltool-debian{a} iptables{a} libabsl20230802{a} libaom3{a} libarchive-zip-perl{a} libavif16{a} libbrotli1{a} libbsd0{a} libcairo2{a} libcdt5{a} libcgraph6{a} libdatrie1{a} libdav1d7{a} libde265-0{a} libdebhelper-perl{a} libdeflate0{a} libdevmapper1.02.1{a} libelf1t64{a} libexpat1{a} libfile-stripnondeterminism-perl{a} libfontconfig1{a} libfreetype6{a} libfribidi0{a} libgav1-1{a} libgd3{a} libglib2.0-0t64{a} libgraphite2-3{a} libgts-0.7-5t64{a} libgvc6{a} libgvpr2{a} libharfbuzz0b{a} libheif-plugin-dav1d{a} libheif-plugin-libde265{a} libheif1{a} libice6{a} libicu72{a} libip4tc2{a} libip6tc2{a} libjbig0{a} libjpeg62-turbo{a} libjs-bootstrap{a} libjs-jquery{a} libjs-sphinxdoc{a} libjs-underscore{a} libjson-perl{a} liblab-gamut1{a} liblerc4{a} libltdl7{a} libmagic-mgc{a} libmagic1t64{a} libmnl0{a} libnetfilter-conntrack3{a} libnfnetlink0{a} libnftnl11{a} libpango-1.0-0{a} libpangocairo-1.0-0{a} libpangoft2-1.0-0{a} libpathplan4{a} libpipeline1{a} libpixman-1-0{a} libpng16-16t64{a} libpopt0{a} libpython3-stdlib{a} libpython3.11-minimal{a} libpython3.11-stdlib{a} libpython3.12-minimal{a} libpython3.12-stdlib{a} librav1e0{a} libreadline8t64{a} libsharpyuv0{a} libsm6{a} libsub-override-perl{a} libsvtav1enc1d1{a} libthai-data{a} libthai0{a} libtiff6{a} libtool{a} libuchardet0{a} libwebp7{a} libx11-6{a} libx11-data{a} libxau6{a} libxaw7{a} libxcb-render0{a} libxcb-shm0{a} libxcb1{a} libxdmcp6{a} libxext6{a} libxml2{a} libxmu6{a} libxpm4{a} libxrender1{a} libxslt1.1{a} libxt6t64{a} libxtables12{a} libyaml-0-2{a} libyuv0{a} m4{a} man-db{a} media-types{a} netbase{a} openssl{a} po-debconf{a} python-babel-localedata{a} python3{a} python3-alabaster{a} python3-all{a} python3-antlr4{a} python3-arcp{a} python3-argcomplete{a} python3-astroid{a} python3-babel{a} python3-blessed{a} python3-boto3{a} python3-botocore{a} python3-cachecontrol{a} python3-certifi{a} python3-chardet{a} python3-charset-normalizer{a} python3-coloredlogs{a} python3-configargparse{a} python3-cwl-utils{a} python3-dateutil{a} python3-dill{a} python3-distutils{a} python3-doc{a} python3-docker{a} python3-docutils{a} python3-enlighten{a} python3-execnet{a} python3-filelock{a} python3-humanfriendly{a} python3-idna{a} python3-imagesize{a} python3-iniconfig{a} python3-isodate{a} python3-jinja2{a} python3-jmespath{a} python3-lib2to3{a} python3-lxml{a} python3-markupsafe{a} python3-minimal{a} python3-mistune{a} python3-msgpack{a} python3-mypy-extensions{a} python3-networkx{a} python3-packaging{a} python3-pkg-resources{a} python3-pluggy{a} python3-prefixed{a} python3-prov{a} python3-psutil{a} python3-pubsub{a} python3-pydot{a} python3-pygments{a} python3-pyparsing{a} python3-pytest{a} python3-pytest-xdist{a} python3-rdflib{a} python3-requests{a} python3-roman{a} python3-ruamel.yaml{a} python3-ruamel.yaml.clib{a} python3-s3transfer{a} python3-schema-salad{a} python3-setuptools{a} python3-shellescape{a} python3-six{a} python3-snowballstemmer{a} python3-sphinx{a} python3-sphinx-autoapi{a} python3-sphinx-autodoc-typehints{a} python3-sphinxcontrib.autoprogram{a} python3-spython{a} python3-stubserver{a} python3-typing-extensions{a} python3-tz{a} python3-unidecode{a} python3-urllib3{a} python3-wcwidth{a} python3-wdlparse{a} python3-websocket{a} python3-yaml{a} python3.11{a} python3.11-doc{a} python3.11-minimal{a} python3.12{a} python3.12-minimal{a} readline-common{a} rsync{a} runc{a} sensible-utils{a} sgml-base{a} sphinx-common{a} tini{a} tzdata{a} x11-common{a} xml-core{a} The following packages are RECOMMENDED but will NOT be installed: apparmor black cgroupfs-mount criu curl fonts-liberation2 git javascript-common libarchive-cpio-perl libglib2.0-data libgts-bin libheif-plugin-aomenc libheif-plugin-x265 libjson-xs-perl libltdl-dev libmail-sendmail-perl libpaper-utils lynx needrestart nftables nodejs python3-bs4 python3-cssselect python3-gdal python3-html5lib python3-matplotlib python3-numpy python3-pil python3-pygraphviz python3-scipy python3-sparqlwrapper shared-mime-info wget xdg-user-dirs 0 packages upgraded, 225 newly installed, 0 to remove and 0 not upgraded. Need to get 149 MB of archives. After unpacking 707 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian unstable/main armhf dmsetup armhf 2:1.02.196-1+b1 [82.3 kB] Get: 2 http://deb.debian.org/debian unstable/main armhf libdevmapper1.02.1 armhf 2:1.02.196-1+b1 [126 kB] Get: 3 http://deb.debian.org/debian unstable/main armhf runc armhf 1.1.12+ds1-2 [2396 kB] Get: 4 http://deb.debian.org/debian unstable/main armhf containerd armhf 1.6.24~ds1-1 [16.0 MB] Get: 5 http://deb.debian.org/debian unstable/main armhf libip4tc2 armhf 1.8.10-3 [17.3 kB] Get: 6 http://deb.debian.org/debian unstable/main armhf libip6tc2 armhf 1.8.10-3 [17.6 kB] Get: 7 http://deb.debian.org/debian unstable/main armhf libxtables12 armhf 1.8.10-3 [28.1 kB] Get: 8 http://deb.debian.org/debian unstable/main armhf netbase all 6.4 [12.8 kB] Get: 9 http://deb.debian.org/debian unstable/main armhf libmnl0 armhf 1.0.5-2+b1 [10.6 kB] Get: 10 http://deb.debian.org/debian unstable/main armhf libnfnetlink0 armhf 1.0.2-3 [12.6 kB] Get: 11 http://deb.debian.org/debian unstable/main armhf libnetfilter-conntrack3 armhf 1.0.9-6+b1 [34.3 kB] Get: 12 http://deb.debian.org/debian unstable/main armhf libnftnl11 armhf 1.2.6-2+b1 [48.4 kB] Get: 13 http://deb.debian.org/debian unstable/main armhf iptables armhf 1.8.10-3 [319 kB] Get: 14 http://deb.debian.org/debian unstable/main armhf tini armhf 0.19.0-1 [188 kB] Get: 15 http://deb.debian.org/debian unstable/main armhf docker.io armhf 20.10.25+dfsg1-2+b3 [28.5 MB] Get: 16 http://deb.debian.org/debian unstable/main armhf libpython3.11-minimal armhf 3.11.9-1 [805 kB] Get: 17 http://deb.debian.org/debian unstable/main armhf libexpat1 armhf 2.6.2-1 [83.5 kB] Get: 18 http://deb.debian.org/debian unstable/main armhf python3.11-minimal armhf 3.11.9-1 [1600 kB] Get: 19 http://deb.debian.org/debian unstable/main armhf python3-minimal armhf 3.11.8-1 [26.3 kB] Get: 20 http://deb.debian.org/debian unstable/main armhf media-types all 10.1.0 [26.9 kB] Get: 21 http://deb.debian.org/debian unstable/main armhf tzdata all 2024a-3 [255 kB] Get: 22 http://deb.debian.org/debian unstable/main armhf readline-common all 8.2-4 [69.3 kB] Get: 23 http://deb.debian.org/debian unstable/main armhf libreadline8t64 armhf 8.2-4 [145 kB] Get: 24 http://deb.debian.org/debian unstable/main armhf libpython3.11-stdlib armhf 3.11.9-1 [1704 kB] Get: 25 http://deb.debian.org/debian unstable/main armhf python3.11 armhf 3.11.9-1 [602 kB] Get: 26 http://deb.debian.org/debian unstable/main armhf libpython3-stdlib armhf 3.11.8-1 [9332 B] Get: 27 http://deb.debian.org/debian unstable/main armhf python3 armhf 3.11.8-1 [27.4 kB] Get: 28 http://deb.debian.org/debian unstable/main armhf libpython3.12-minimal armhf 3.12.3-1 [795 kB] Get: 29 http://deb.debian.org/debian unstable/main armhf python3.12-minimal armhf 3.12.3-1 [1783 kB] Get: 30 http://deb.debian.org/debian unstable/main armhf libpopt0 armhf 1.19+dfsg-1+b1 [40.6 kB] Get: 31 http://deb.debian.org/debian unstable/main armhf rsync armhf 3.3.0-1 [398 kB] Get: 32 http://deb.debian.org/debian unstable/main armhf sgml-base all 1.31 [15.4 kB] Get: 33 http://deb.debian.org/debian unstable/main armhf sensible-utils all 0.0.22 [22.4 kB] Get: 34 http://deb.debian.org/debian unstable/main armhf openssl armhf 3.2.1-3 [1326 kB] Get: 35 http://deb.debian.org/debian unstable/main armhf ca-certificates all 20240203 [158 kB] Get: 36 http://deb.debian.org/debian unstable/main armhf libmagic-mgc armhf 1:5.45-3 [314 kB] Get: 37 http://deb.debian.org/debian unstable/main armhf libmagic1t64 armhf 1:5.45-3 [98.1 kB] Get: 38 http://deb.debian.org/debian unstable/main armhf file armhf 1:5.45-3 [42.0 kB] Get: 39 http://deb.debian.org/debian unstable/main armhf gettext-base armhf 0.21-14+b1 [157 kB] Get: 40 http://deb.debian.org/debian unstable/main armhf libuchardet0 armhf 0.0.8-1+b1 [65.7 kB] Get: 41 http://deb.debian.org/debian unstable/main armhf groff-base armhf 1.23.0-4 [1090 kB] Get: 42 http://deb.debian.org/debian unstable/main armhf bsdextrautils armhf 2.40-8 [85.6 kB] Get: 43 http://deb.debian.org/debian unstable/main armhf libpipeline1 armhf 1.5.7-2 [33.3 kB] Get: 44 http://deb.debian.org/debian unstable/main armhf man-db armhf 2.12.1-1 [1375 kB] Get: 45 http://deb.debian.org/debian unstable/main armhf m4 armhf 1.4.19-4 [264 kB] Get: 46 http://deb.debian.org/debian unstable/main armhf autoconf all 2.71-3 [332 kB] Get: 47 http://deb.debian.org/debian unstable/main armhf autotools-dev all 20220109.1 [51.6 kB] Get: 48 http://deb.debian.org/debian unstable/main armhf automake all 1:1.16.5-1.3 [823 kB] Get: 49 http://deb.debian.org/debian unstable/main armhf autopoint all 0.21-14 [496 kB] Get: 50 http://deb.debian.org/debian unstable/main armhf python3-pkg-resources all 68.1.2-2 [241 kB] Get: 51 http://deb.debian.org/debian unstable/main armhf python3-ruamel.yaml.clib armhf 0.2.8+ds-1 [160 kB] Get: 52 http://deb.debian.org/debian unstable/main armhf python3-ruamel.yaml all 0.18.6+ds-3 [145 kB] Get: 53 http://deb.debian.org/debian unstable/main armhf python3-certifi all 2023.11.17-1 [155 kB] Get: 54 http://deb.debian.org/debian unstable/main armhf python3-charset-normalizer all 3.3.2-1 [51.6 kB] Get: 55 http://deb.debian.org/debian unstable/main armhf python3-idna all 3.6-2 [37.0 kB] Get: 56 http://deb.debian.org/debian unstable/main armhf python3-six all 1.16.0-6 [16.3 kB] Get: 57 http://deb.debian.org/debian unstable/main armhf python3-urllib3 all 1.26.18-2 [116 kB] Get: 58 http://deb.debian.org/debian unstable/main armhf python3-chardet all 5.2.0+dfsg-1 [107 kB] Get: 59 http://deb.debian.org/debian unstable/main armhf python3-requests all 2.31.0+dfsg-1 [68.6 kB] Get: 60 http://deb.debian.org/debian unstable/main armhf python3-msgpack armhf 1.0.3-3+b1 [89.1 kB] Get: 61 http://deb.debian.org/debian unstable/main armhf python3-cachecontrol all 0.14.0-1 [18.6 kB] Get: 62 http://deb.debian.org/debian unstable/main armhf python3-mistune all 3.0.2-1 [33.1 kB] Get: 63 http://deb.debian.org/debian unstable/main armhf python3-mypy-extensions all 1.0.0-1 [6064 B] Get: 64 http://deb.debian.org/debian unstable/main armhf python3-isodate all 0.6.1-2 [24.9 kB] Get: 65 http://deb.debian.org/debian unstable/main armhf python3-pyparsing all 3.1.2-1 [146 kB] Get: 66 http://deb.debian.org/debian unstable/main armhf python3-rdflib all 6.3.2-1 [395 kB] Get: 67 http://deb.debian.org/debian unstable/main armhf python3-filelock all 3.13.4-1 [10.8 kB] Get: 68 http://deb.debian.org/debian unstable/main armhf fonts-glyphicons-halflings all 1.009~3.4.1+dfsg-3 [162 kB] Get: 69 http://deb.debian.org/debian unstable/main armhf libjs-bootstrap all 3.4.1+dfsg-3 [172 kB] Get: 70 http://deb.debian.org/debian unstable/main armhf python3-schema-salad armhf 8.5.20240311110950-1 [1452 kB] Get: 71 http://deb.debian.org/debian unstable/main armhf cwl-upgrader all 1.2.11-1 [15.7 kB] Get: 72 http://deb.debian.org/debian unstable/main armhf python3-argcomplete all 3.3.0-1 [38.9 kB] Get: 73 http://deb.debian.org/debian unstable/main armhf python3-humanfriendly all 10.0-5 [85.4 kB] Get: 74 http://deb.debian.org/debian unstable/main armhf python3-coloredlogs all 15.0.1-1 [47.4 kB] Get: 75 http://deb.debian.org/debian unstable/main armhf python3-packaging all 24.0-1 [45.5 kB] Get: 76 http://deb.debian.org/debian unstable/main armhf python3-cwl-utils all 0.33-1 [118 kB] Get: 77 http://deb.debian.org/debian unstable/main armhf python3-dateutil all 2.9.0-2 [79.4 kB] Get: 78 http://deb.debian.org/debian unstable/main armhf libicu72 armhf 72.1-4+b1 [9070 kB] Get: 79 http://deb.debian.org/debian unstable/main armhf libxml2 armhf 2.9.14+dfsg-1.3+b3 [598 kB] Get: 80 http://deb.debian.org/debian unstable/main armhf libxslt1.1 armhf 1.1.35-1+b1 [212 kB] Get: 81 http://deb.debian.org/debian unstable/main armhf python3-lxml armhf 5.2.1-1 [1788 kB] Get: 82 http://deb.debian.org/debian unstable/main armhf python3-networkx all 3.2.1-3 [14.1 MB] Get: 83 http://deb.debian.org/debian unstable/main armhf python3-prov all 2.0.0-2 [95.6 kB] Get: 84 http://deb.debian.org/debian unstable/main armhf python3-psutil armhf 5.9.8-2 [225 kB] Get: 85 http://deb.debian.org/debian unstable/main armhf libcdt5 armhf 2.42.2-9+b1 [37.3 kB] Get: 86 http://deb.debian.org/debian unstable/main armhf libcgraph6 armhf 2.42.2-9+b1 [56.3 kB] Get: 87 http://deb.debian.org/debian unstable/main armhf libaom3 armhf 3.8.2-2 [1526 kB] Get: 88 http://deb.debian.org/debian unstable/main armhf libdav1d7 armhf 1.4.1-1 [262 kB] Get: 89 http://deb.debian.org/debian unstable/main armhf libabsl20230802 armhf 20230802.1-4 [432 kB] Get: 90 http://deb.debian.org/debian unstable/main armhf libgav1-1 armhf 0.19.0-2+b1 [299 kB] Get: 91 http://deb.debian.org/debian unstable/main armhf librav1e0 armhf 0.7.1-2 [546 kB] Get: 92 http://deb.debian.org/debian unstable/main armhf libsharpyuv0 armhf 1.3.2-0.4+b1 [106 kB] Get: 93 http://deb.debian.org/debian unstable/main armhf libsvtav1enc1d1 armhf 1.7.0+dfsg-2+b1 [936 kB] Get: 94 http://deb.debian.org/debian unstable/main armhf libjpeg62-turbo armhf 1:2.1.5-3 [143 kB] Get: 95 http://deb.debian.org/debian unstable/main armhf libyuv0 armhf 0.0~git202401110.af6ac82-1 [85.1 kB] Get: 96 http://deb.debian.org/debian unstable/main armhf libavif16 armhf 1.0.4-2 [94.9 kB] Get: 97 http://deb.debian.org/debian unstable/main armhf libbrotli1 armhf 1.1.0-2+b3 [284 kB] Get: 98 http://deb.debian.org/debian unstable/main armhf libpng16-16t64 armhf 1.6.43-5 [262 kB] Get: 99 http://deb.debian.org/debian unstable/main armhf libfreetype6 armhf 2.13.2+dfsg-1+b4 [372 kB] Get: 100 http://deb.debian.org/debian unstable/main armhf fonts-dejavu-mono all 2.37-8 [489 kB] Get: 101 http://deb.debian.org/debian unstable/main armhf fonts-dejavu-core all 2.37-8 [840 kB] Get: 102 http://deb.debian.org/debian unstable/main armhf fontconfig-config armhf 2.15.0-1.1 [317 kB] Get: 103 http://deb.debian.org/debian unstable/main armhf libfontconfig1 armhf 2.15.0-1.1 [370 kB] Get: 104 http://deb.debian.org/debian unstable/main armhf libheif-plugin-dav1d armhf 1.17.6-1+b2 [9508 B] Get: 105 http://deb.debian.org/debian unstable/main armhf libde265-0 armhf 1.0.15-1+b1 [145 kB] Get: 106 http://deb.debian.org/debian unstable/main armhf libheif-plugin-libde265 armhf 1.17.6-1+b2 [12.3 kB] Get: 107 http://deb.debian.org/debian unstable/main armhf libheif1 armhf 1.17.6-1+b2 [243 kB] Get: 108 http://deb.debian.org/debian unstable/main armhf libdeflate0 armhf 1.20-1 [35.9 kB] Get: 109 http://deb.debian.org/debian unstable/main armhf libjbig0 armhf 2.1-6.1+b1 [27.3 kB] Get: 110 http://deb.debian.org/debian unstable/main armhf liblerc4 armhf 4.0.0+ds-4+b1 [137 kB] Get: 111 http://deb.debian.org/debian unstable/main armhf libwebp7 armhf 1.3.2-0.4+b1 [262 kB] Get: 112 http://deb.debian.org/debian unstable/main armhf libtiff6 armhf 4.5.1+git230720-4 [301 kB] Get: 113 http://deb.debian.org/debian unstable/main armhf libxau6 armhf 1:1.0.9-1+b1 [17.4 kB] Get: 114 http://deb.debian.org/debian unstable/main armhf libbsd0 armhf 0.12.2-1 [127 kB] Get: 115 http://deb.debian.org/debian unstable/main armhf libxdmcp6 armhf 1:1.1.2-3+b1 [23.0 kB] Get: 116 http://deb.debian.org/debian unstable/main armhf libxcb1 armhf 1.17.0-1 [140 kB] Get: 117 http://deb.debian.org/debian unstable/main armhf libx11-data all 2:1.8.7-1 [328 kB] Get: 118 http://deb.debian.org/debian unstable/main armhf libx11-6 armhf 2:1.8.7-1+b1 [739 kB] Get: 119 http://deb.debian.org/debian unstable/main armhf libxpm4 armhf 1:3.5.17-1+b1 [50.0 kB] Get: 120 http://deb.debian.org/debian unstable/main armhf libgd3 armhf 2.3.3-9+b3 [104 kB] Get: 121 http://deb.debian.org/debian unstable/main armhf libglib2.0-0t64 armhf 2.78.4-7 [1285 kB] Get: 122 http://deb.debian.org/debian unstable/main armhf libgts-0.7-5t64 armhf 0.7.6+darcs121130-5.2 [132 kB] Get: 123 http://deb.debian.org/debian unstable/main armhf libpixman-1-0 armhf 0.42.2-1+b1 [476 kB] Get: 124 http://deb.debian.org/debian unstable/main armhf libxcb-render0 armhf 1.17.0-1 [114 kB] Get: 125 http://deb.debian.org/debian unstable/main armhf libxcb-shm0 armhf 1.17.0-1 [105 kB] Get: 126 http://deb.debian.org/debian unstable/main armhf libxext6 armhf 2:1.3.4-1+b1 [47.8 kB] Get: 127 http://deb.debian.org/debian unstable/main armhf libxrender1 armhf 1:0.9.10-1.1+b1 [24.9 kB] Get: 128 http://deb.debian.org/debian unstable/main armhf libcairo2 armhf 1.18.0-3+b1 [442 kB] Get: 129 http://deb.debian.org/debian unstable/main armhf libltdl7 armhf 2.4.7-7+b1 [390 kB] Get: 130 http://deb.debian.org/debian unstable/main armhf fontconfig armhf 2.15.0-1.1 [461 kB] Get: 131 http://deb.debian.org/debian unstable/main armhf libfribidi0 armhf 1.0.13-3+b1 [69.4 kB] Get: 132 http://deb.debian.org/debian unstable/main armhf libgraphite2-3 armhf 1.3.14-2 [63.2 kB] Get: 133 http://deb.debian.org/debian unstable/main armhf libharfbuzz0b armhf 8.3.0-2+b1 [2156 kB] Get: 134 http://deb.debian.org/debian unstable/main armhf libthai-data all 0.1.29-2 [168 kB] Get: 135 http://deb.debian.org/debian unstable/main armhf libdatrie1 armhf 0.2.13-3 [34.4 kB] Get: 136 http://deb.debian.org/debian unstable/main armhf libthai0 armhf 0.1.29-2 [45.8 kB] Get: 137 http://deb.debian.org/debian unstable/main armhf libpango-1.0-0 armhf 1.52.1+ds-1 [192 kB] Get: 138 http://deb.debian.org/debian unstable/main armhf libpangoft2-1.0-0 armhf 1.52.1+ds-1 [41.8 kB] Get: 139 http://deb.debian.org/debian unstable/main armhf libpangocairo-1.0-0 armhf 1.52.1+ds-1 [31.2 kB] Get: 140 http://deb.debian.org/debian unstable/main armhf libpathplan4 armhf 2.42.2-9+b1 [39.7 kB] Get: 141 http://deb.debian.org/debian unstable/main armhf libgvc6 armhf 2.42.2-9+b1 [578 kB] Get: 142 http://deb.debian.org/debian unstable/main armhf libgvpr2 armhf 2.42.2-9+b1 [165 kB] Get: 143 http://deb.debian.org/debian unstable/main armhf liblab-gamut1 armhf 2.42.2-9+b1 [198 kB] Get: 144 http://deb.debian.org/debian unstable/main armhf x11-common all 1:7.7+23 [252 kB] Get: 145 http://deb.debian.org/debian unstable/main armhf libice6 armhf 2:1.0.10-1+b1 [50.1 kB] Get: 146 http://deb.debian.org/debian unstable/main armhf libsm6 armhf 2:1.2.3-1+b1 [31.7 kB] Get: 147 http://deb.debian.org/debian unstable/main armhf libxt6t64 armhf 1:1.2.1-1.2 [159 kB] Get: 148 http://deb.debian.org/debian unstable/main armhf libxmu6 armhf 2:1.1.3-3+b2 [50.9 kB] Get: 149 http://deb.debian.org/debian unstable/main armhf libxaw7 armhf 2:1.0.14-1+b2 [165 kB] Get: 150 http://deb.debian.org/debian unstable/main armhf graphviz armhf 2.42.2-9+b1 [499 kB] Get: 151 http://deb.debian.org/debian unstable/main armhf python3-pydot all 2.0.0-2 [32.5 kB] Get: 152 http://deb.debian.org/debian unstable/main armhf python3-shellescape all 3.4.1-5 [5040 B] Get: 153 http://deb.debian.org/debian unstable/main armhf python3-spython all 0.3.13-2 [55.2 kB] Get: 154 http://deb.debian.org/debian unstable/main armhf python3-arcp all 0.2.1-6 [13.5 kB] Get: 155 http://deb.debian.org/debian unstable/main armhf python3-lib2to3 all 3.12.3-1 [77.6 kB] Get: 156 http://deb.debian.org/debian unstable/main armhf python3-distutils all 3.12.3-1 [131 kB] Get: 157 http://deb.debian.org/debian unstable/main armhf python3-setuptools all 68.1.2-2 [468 kB] Get: 158 http://deb.debian.org/debian unstable/main armhf cwltool all 3.1.20240404144621-2 [490 kB] Get: 159 http://deb.debian.org/debian unstable/main armhf libdebhelper-perl all 13.15.3 [88.0 kB] Get: 160 http://deb.debian.org/debian unstable/main armhf libtool all 2.4.7-7 [517 kB] Get: 161 http://deb.debian.org/debian unstable/main armhf dh-autoreconf all 20 [17.1 kB] Get: 162 http://deb.debian.org/debian unstable/main armhf libarchive-zip-perl all 1.68-1 [104 kB] Get: 163 http://deb.debian.org/debian unstable/main armhf libsub-override-perl all 0.10-1 [10.6 kB] Get: 164 http://deb.debian.org/debian unstable/main armhf libfile-stripnondeterminism-perl all 1.13.1-1 [19.4 kB] Get: 165 http://deb.debian.org/debian unstable/main armhf dh-strip-nondeterminism all 1.13.1-1 [8620 B] Get: 166 http://deb.debian.org/debian unstable/main armhf libelf1t64 armhf 0.191-1+b1 [183 kB] Get: 167 http://deb.debian.org/debian unstable/main armhf dwz armhf 0.15-1+b2 [106 kB] Get: 168 http://deb.debian.org/debian unstable/main armhf gettext armhf 0.21-14+b1 [1230 kB] Get: 169 http://deb.debian.org/debian unstable/main armhf intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 170 http://deb.debian.org/debian unstable/main armhf po-debconf all 1.0.21+nmu1 [248 kB] Get: 171 http://deb.debian.org/debian unstable/main armhf debhelper all 13.15.3 [901 kB] Get: 172 http://deb.debian.org/debian unstable/main armhf dh-python all 6.20240422 [107 kB] Get: 173 http://deb.debian.org/debian unstable/main armhf xml-core all 0.19 [20.1 kB] Get: 174 http://deb.debian.org/debian unstable/main armhf docutils-common all 0.20.1+dfsg-3 [128 kB] Get: 175 http://deb.debian.org/debian unstable/main armhf libjs-jquery all 3.6.1+dfsg+~3.5.14-1 [326 kB] Get: 176 http://deb.debian.org/debian unstable/main armhf libjs-underscore all 1.13.4~dfsg+~1.11.4-3 [116 kB] Get: 177 http://deb.debian.org/debian unstable/main armhf libjs-sphinxdoc all 7.2.6-6 [150 kB] Get: 178 http://deb.debian.org/debian unstable/main armhf libjson-perl all 4.10000-1 [87.5 kB] Get: 179 http://deb.debian.org/debian unstable/main armhf libpython3.12-stdlib armhf 3.12.3-1 [1808 kB] Get: 180 http://deb.debian.org/debian unstable/main armhf libyaml-0-2 armhf 0.2.5-1+b1 [45.5 kB] Get: 181 http://deb.debian.org/debian unstable/main armhf python-babel-localedata all 2.14.0-1 [5701 kB] Get: 182 http://deb.debian.org/debian unstable/main armhf python3-alabaster all 0.7.12-1 [20.8 kB] Get: 183 http://deb.debian.org/debian unstable/main armhf python3.12 armhf 3.12.3-1 [659 kB] Get: 184 http://deb.debian.org/debian unstable/main armhf python3-all armhf 3.11.8-1 [1056 B] Get: 185 http://deb.debian.org/debian unstable/main armhf python3-antlr4 all 4.9.2-1 [98.1 kB] Get: 186 http://deb.debian.org/debian unstable/main armhf python3-typing-extensions all 4.10.0-1 [66.2 kB] Get: 187 http://deb.debian.org/debian unstable/main armhf python3-astroid all 3.1.0-1 [211 kB] Get: 188 http://deb.debian.org/debian unstable/main armhf python3-tz all 2024.1-2 [30.9 kB] Get: 189 http://deb.debian.org/debian unstable/main armhf python3-babel all 2.14.0-1 [111 kB] Get: 190 http://deb.debian.org/debian unstable/main armhf python3-wcwidth all 0.2.13+dfsg1-1 [24.9 kB] Get: 191 http://deb.debian.org/debian unstable/main armhf python3-blessed all 1.20.0-1 [54.2 kB] Get: 192 http://deb.debian.org/debian unstable/main armhf python3-jmespath all 1.0.1-1 [21.1 kB] Get: 193 http://deb.debian.org/debian unstable/main armhf python3-botocore all 1.34.46+repack-1 [6464 kB] Get: 194 http://deb.debian.org/debian unstable/main armhf python3-s3transfer all 0.10.1-1 [55.6 kB] Get: 195 http://deb.debian.org/debian unstable/main armhf python3-boto3 all 1.34.46+dfsg-1 [269 kB] Get: 196 http://deb.debian.org/debian unstable/main armhf python3-configargparse all 1.7-1 [31.3 kB] Get: 197 http://deb.debian.org/debian unstable/main armhf python3-dill all 0.3.8-1 [81.7 kB] Get: 198 http://deb.debian.org/debian unstable/main armhf python3.11-doc all 3.11.9-1 [12.6 MB] Get: 199 http://deb.debian.org/debian unstable/main armhf python3-doc all 3.11.8-1 [9528 B] Get: 200 http://deb.debian.org/debian unstable/main armhf python3-websocket all 1.7.0-1 [45.0 kB] Get: 201 http://deb.debian.org/debian unstable/main armhf python3-docker all 6.1.3-0.1 [89.2 kB] Get: 202 http://deb.debian.org/debian unstable/main armhf python3-roman all 3.3-3 [9880 B] Get: 203 http://deb.debian.org/debian unstable/main armhf python3-docutils all 0.20.1+dfsg-3 [389 kB] Get: 204 http://deb.debian.org/debian unstable/main armhf python3-prefixed all 0.7.0-1 [7984 B] Get: 205 http://deb.debian.org/debian unstable/main armhf python3-enlighten armhf 1.11.2-1 [41.4 kB] Get: 206 http://deb.debian.org/debian unstable/main armhf python3-execnet all 2.1.1-1 [40.2 kB] Get: 207 http://deb.debian.org/debian unstable/main armhf python3-imagesize all 1.4.1-1 [6688 B] Get: 208 http://deb.debian.org/debian unstable/main armhf python3-iniconfig all 1.1.1-2 [6396 B] Get: 209 http://deb.debian.org/debian unstable/main armhf python3-markupsafe armhf 2.1.5-1 [13.9 kB] Get: 210 http://deb.debian.org/debian unstable/main armhf python3-jinja2 all 3.1.3-1 [119 kB] Get: 211 http://deb.debian.org/debian unstable/main armhf python3-pluggy all 1.5.0-1 [26.9 kB] Get: 212 http://deb.debian.org/debian unstable/main armhf python3-pubsub all 4.0.3-7 [46.7 kB] Get: 213 http://deb.debian.org/debian unstable/main armhf python3-pygments all 2.17.2+dfsg-1 [818 kB] Get: 214 http://deb.debian.org/debian unstable/main armhf python3-pytest all 8.1.2-1 [246 kB] Get: 215 http://deb.debian.org/debian unstable/main armhf python3-pytest-xdist all 3.5.0-1 [42.1 kB] Get: 216 http://deb.debian.org/debian unstable/main armhf python3-snowballstemmer all 2.2.0-4 [58.0 kB] Get: 217 http://deb.debian.org/debian unstable/main armhf sphinx-common all 7.2.6-6 [702 kB] Get: 218 http://deb.debian.org/debian unstable/main armhf python3-sphinx all 7.2.6-6 [552 kB] Get: 219 http://deb.debian.org/debian unstable/main armhf python3-unidecode all 1.3.8-1 [115 kB] Get: 220 http://deb.debian.org/debian unstable/main armhf python3-yaml armhf 6.0.1-2 [162 kB] Get: 221 http://deb.debian.org/debian unstable/main armhf python3-sphinx-autoapi all 3.0.0-0.1 [103 kB] Get: 222 http://deb.debian.org/debian unstable/main armhf python3-sphinx-autodoc-typehints all 1.25.3-1 [24.4 kB] Get: 223 http://deb.debian.org/debian unstable/main armhf python3-sphinxcontrib.autoprogram all 0.1.8-2 [10.6 kB] Get: 224 http://deb.debian.org/debian unstable/main armhf python3-stubserver all 1.1-4 [7764 B] Get: 225 http://deb.debian.org/debian unstable/main armhf python3-wdlparse all 0.1.0-3 [66.1 kB] Fetched 149 MB in 17s (8835 kB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package dmsetup. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19440 files and directories currently installed.) Preparing to unpack .../00-dmsetup_2%3a1.02.196-1+b1_armhf.deb ... Unpacking dmsetup (2:1.02.196-1+b1) ... Selecting previously unselected package libdevmapper1.02.1:armhf. Preparing to unpack .../01-libdevmapper1.02.1_2%3a1.02.196-1+b1_armhf.deb ... Unpacking libdevmapper1.02.1:armhf (2:1.02.196-1+b1) ... Selecting previously unselected package runc. Preparing to unpack .../02-runc_1.1.12+ds1-2_armhf.deb ... Unpacking runc (1.1.12+ds1-2) ... Selecting previously unselected package containerd. Preparing to unpack .../03-containerd_1.6.24~ds1-1_armhf.deb ... Unpacking containerd (1.6.24~ds1-1) ... Selecting previously unselected package libip4tc2:armhf. Preparing to unpack .../04-libip4tc2_1.8.10-3_armhf.deb ... Unpacking libip4tc2:armhf (1.8.10-3) ... Selecting previously unselected package libip6tc2:armhf. Preparing to unpack .../05-libip6tc2_1.8.10-3_armhf.deb ... Unpacking libip6tc2:armhf (1.8.10-3) ... Selecting previously unselected package libxtables12:armhf. Preparing to unpack .../06-libxtables12_1.8.10-3_armhf.deb ... Unpacking libxtables12:armhf (1.8.10-3) ... Selecting previously unselected package netbase. Preparing to unpack .../07-netbase_6.4_all.deb ... Unpacking netbase (6.4) ... Selecting previously unselected package libmnl0:armhf. Preparing to unpack .../08-libmnl0_1.0.5-2+b1_armhf.deb ... Unpacking libmnl0:armhf (1.0.5-2+b1) ... Selecting previously unselected package libnfnetlink0:armhf. Preparing to unpack .../09-libnfnetlink0_1.0.2-3_armhf.deb ... Unpacking libnfnetlink0:armhf (1.0.2-3) ... Selecting previously unselected package libnetfilter-conntrack3:armhf. Preparing to unpack .../10-libnetfilter-conntrack3_1.0.9-6+b1_armhf.deb ... Unpacking libnetfilter-conntrack3:armhf (1.0.9-6+b1) ... Selecting previously unselected package libnftnl11:armhf. Preparing to unpack .../11-libnftnl11_1.2.6-2+b1_armhf.deb ... Unpacking libnftnl11:armhf (1.2.6-2+b1) ... Selecting previously unselected package iptables. Preparing to unpack .../12-iptables_1.8.10-3_armhf.deb ... Unpacking iptables (1.8.10-3) ... Selecting previously unselected package tini. Preparing to unpack .../13-tini_0.19.0-1_armhf.deb ... Unpacking tini (0.19.0-1) ... Selecting previously unselected package docker.io. Preparing to unpack .../14-docker.io_20.10.25+dfsg1-2+b3_armhf.deb ... Unpacking docker.io (20.10.25+dfsg1-2+b3) ... Selecting previously unselected package libpython3.11-minimal:armhf. Preparing to unpack .../15-libpython3.11-minimal_3.11.9-1_armhf.deb ... Unpacking libpython3.11-minimal:armhf (3.11.9-1) ... Selecting previously unselected package libexpat1:armhf. Preparing to unpack .../16-libexpat1_2.6.2-1_armhf.deb ... Unpacking libexpat1:armhf (2.6.2-1) ... Selecting previously unselected package python3.11-minimal. Preparing to unpack .../17-python3.11-minimal_3.11.9-1_armhf.deb ... Unpacking python3.11-minimal (3.11.9-1) ... Setting up libpython3.11-minimal:armhf (3.11.9-1) ... Setting up libexpat1:armhf (2.6.2-1) ... Setting up python3.11-minimal (3.11.9-1) ... Selecting previously unselected package python3-minimal. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 20321 files and directories currently installed.) Preparing to unpack .../0-python3-minimal_3.11.8-1_armhf.deb ... Unpacking python3-minimal (3.11.8-1) ... Selecting previously unselected package media-types. Preparing to unpack .../1-media-types_10.1.0_all.deb ... Unpacking media-types (10.1.0) ... Selecting previously unselected package tzdata. Preparing to unpack .../2-tzdata_2024a-3_all.deb ... Unpacking tzdata (2024a-3) ... Selecting previously unselected package readline-common. Preparing to unpack .../3-readline-common_8.2-4_all.deb ... Unpacking readline-common (8.2-4) ... Selecting previously unselected package libreadline8t64:armhf. Preparing to unpack .../4-libreadline8t64_8.2-4_armhf.deb ... Adding 'diversion of /lib/arm-linux-gnueabihf/libhistory.so.8 to /lib/arm-linux-gnueabihf/libhistory.so.8.usr-is-merged by libreadline8t64' Adding 'diversion of /lib/arm-linux-gnueabihf/libhistory.so.8.2 to /lib/arm-linux-gnueabihf/libhistory.so.8.2.usr-is-merged by libreadline8t64' Adding 'diversion of /lib/arm-linux-gnueabihf/libreadline.so.8 to /lib/arm-linux-gnueabihf/libreadline.so.8.usr-is-merged by libreadline8t64' Adding 'diversion of /lib/arm-linux-gnueabihf/libreadline.so.8.2 to /lib/arm-linux-gnueabihf/libreadline.so.8.2.usr-is-merged by libreadline8t64' Unpacking libreadline8t64:armhf (8.2-4) ... Selecting previously unselected package libpython3.11-stdlib:armhf. Preparing to unpack .../5-libpython3.11-stdlib_3.11.9-1_armhf.deb ... Unpacking libpython3.11-stdlib:armhf (3.11.9-1) ... Selecting previously unselected package python3.11. Preparing to unpack .../6-python3.11_3.11.9-1_armhf.deb ... Unpacking python3.11 (3.11.9-1) ... Selecting previously unselected package libpython3-stdlib:armhf. Preparing to unpack .../7-libpython3-stdlib_3.11.8-1_armhf.deb ... Unpacking libpython3-stdlib:armhf (3.11.8-1) ... Setting up python3-minimal (3.11.8-1) ... Selecting previously unselected package python3. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 21306 files and directories currently installed.) Preparing to unpack .../000-python3_3.11.8-1_armhf.deb ... Unpacking python3 (3.11.8-1) ... Selecting previously unselected package libpython3.12-minimal:armhf. Preparing to unpack .../001-libpython3.12-minimal_3.12.3-1_armhf.deb ... Unpacking libpython3.12-minimal:armhf (3.12.3-1) ... Selecting previously unselected package python3.12-minimal. Preparing to unpack .../002-python3.12-minimal_3.12.3-1_armhf.deb ... Unpacking python3.12-minimal (3.12.3-1) ... Selecting previously unselected package libpopt0:armhf. Preparing to unpack .../003-libpopt0_1.19+dfsg-1+b1_armhf.deb ... Unpacking libpopt0:armhf (1.19+dfsg-1+b1) ... Selecting previously unselected package rsync. Preparing to unpack .../004-rsync_3.3.0-1_armhf.deb ... Unpacking rsync (3.3.0-1) ... Selecting previously unselected package sgml-base. Preparing to unpack .../005-sgml-base_1.31_all.deb ... Unpacking sgml-base (1.31) ... Selecting previously unselected package sensible-utils. Preparing to unpack .../006-sensible-utils_0.0.22_all.deb ... Unpacking sensible-utils (0.0.22) ... Selecting previously unselected package openssl. Preparing to unpack .../007-openssl_3.2.1-3_armhf.deb ... Unpacking openssl (3.2.1-3) ... Selecting previously unselected package ca-certificates. Preparing to unpack .../008-ca-certificates_20240203_all.deb ... Unpacking ca-certificates (20240203) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../009-libmagic-mgc_1%3a5.45-3_armhf.deb ... Unpacking libmagic-mgc (1:5.45-3) ... Selecting previously unselected package libmagic1t64:armhf. Preparing to unpack .../010-libmagic1t64_1%3a5.45-3_armhf.deb ... Unpacking libmagic1t64:armhf (1:5.45-3) ... Selecting previously unselected package file. Preparing to unpack .../011-file_1%3a5.45-3_armhf.deb ... Unpacking file (1:5.45-3) ... Selecting previously unselected package gettext-base. Preparing to unpack .../012-gettext-base_0.21-14+b1_armhf.deb ... Unpacking gettext-base (0.21-14+b1) ... Selecting previously unselected package libuchardet0:armhf. Preparing to unpack .../013-libuchardet0_0.0.8-1+b1_armhf.deb ... Unpacking libuchardet0:armhf (0.0.8-1+b1) ... Selecting previously unselected package groff-base. Preparing to unpack .../014-groff-base_1.23.0-4_armhf.deb ... Unpacking groff-base (1.23.0-4) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../015-bsdextrautils_2.40-8_armhf.deb ... Unpacking bsdextrautils (2.40-8) ... Selecting previously unselected package libpipeline1:armhf. Preparing to unpack .../016-libpipeline1_1.5.7-2_armhf.deb ... Unpacking libpipeline1:armhf (1.5.7-2) ... Selecting previously unselected package man-db. Preparing to unpack .../017-man-db_2.12.1-1_armhf.deb ... Unpacking man-db (2.12.1-1) ... Selecting previously unselected package m4. Preparing to unpack .../018-m4_1.4.19-4_armhf.deb ... Unpacking m4 (1.4.19-4) ... Selecting previously unselected package autoconf. Preparing to unpack .../019-autoconf_2.71-3_all.deb ... Unpacking autoconf (2.71-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../020-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../021-automake_1%3a1.16.5-1.3_all.deb ... Unpacking automake (1:1.16.5-1.3) ... Selecting previously unselected package autopoint. Preparing to unpack .../022-autopoint_0.21-14_all.deb ... Unpacking autopoint (0.21-14) ... Selecting previously unselected package python3-pkg-resources. Preparing to unpack .../023-python3-pkg-resources_68.1.2-2_all.deb ... Unpacking python3-pkg-resources (68.1.2-2) ... Selecting previously unselected package python3-ruamel.yaml.clib:armhf. Preparing to unpack .../024-python3-ruamel.yaml.clib_0.2.8+ds-1_armhf.deb ... Unpacking python3-ruamel.yaml.clib:armhf (0.2.8+ds-1) ... Selecting previously unselected package python3-ruamel.yaml. Preparing to unpack .../025-python3-ruamel.yaml_0.18.6+ds-3_all.deb ... Unpacking python3-ruamel.yaml (0.18.6+ds-3) ... Selecting previously unselected package python3-certifi. Preparing to unpack .../026-python3-certifi_2023.11.17-1_all.deb ... Unpacking python3-certifi (2023.11.17-1) ... Selecting previously unselected package python3-charset-normalizer. Preparing to unpack .../027-python3-charset-normalizer_3.3.2-1_all.deb ... Unpacking python3-charset-normalizer (3.3.2-1) ... Selecting previously unselected package python3-idna. Preparing to unpack .../028-python3-idna_3.6-2_all.deb ... Unpacking python3-idna (3.6-2) ... Selecting previously unselected package python3-six. Preparing to unpack .../029-python3-six_1.16.0-6_all.deb ... Unpacking python3-six (1.16.0-6) ... Selecting previously unselected package python3-urllib3. Preparing to unpack .../030-python3-urllib3_1.26.18-2_all.deb ... Unpacking python3-urllib3 (1.26.18-2) ... Selecting previously unselected package python3-chardet. Preparing to unpack .../031-python3-chardet_5.2.0+dfsg-1_all.deb ... Unpacking python3-chardet (5.2.0+dfsg-1) ... Selecting previously unselected package python3-requests. Preparing to unpack .../032-python3-requests_2.31.0+dfsg-1_all.deb ... Unpacking python3-requests (2.31.0+dfsg-1) ... Selecting previously unselected package python3-msgpack. Preparing to unpack .../033-python3-msgpack_1.0.3-3+b1_armhf.deb ... Unpacking python3-msgpack (1.0.3-3+b1) ... Selecting previously unselected package python3-cachecontrol. Preparing to unpack .../034-python3-cachecontrol_0.14.0-1_all.deb ... Unpacking python3-cachecontrol (0.14.0-1) ... Selecting previously unselected package python3-mistune. Preparing to unpack .../035-python3-mistune_3.0.2-1_all.deb ... Unpacking python3-mistune (3.0.2-1) ... Selecting previously unselected package python3-mypy-extensions. Preparing to unpack .../036-python3-mypy-extensions_1.0.0-1_all.deb ... Unpacking python3-mypy-extensions (1.0.0-1) ... Selecting previously unselected package python3-isodate. Preparing to unpack .../037-python3-isodate_0.6.1-2_all.deb ... Unpacking python3-isodate (0.6.1-2) ... Selecting previously unselected package python3-pyparsing. Preparing to unpack .../038-python3-pyparsing_3.1.2-1_all.deb ... Unpacking python3-pyparsing (3.1.2-1) ... Selecting previously unselected package python3-rdflib. Preparing to unpack .../039-python3-rdflib_6.3.2-1_all.deb ... Unpacking python3-rdflib (6.3.2-1) ... Selecting previously unselected package python3-filelock. Preparing to unpack .../040-python3-filelock_3.13.4-1_all.deb ... Unpacking python3-filelock (3.13.4-1) ... Selecting previously unselected package fonts-glyphicons-halflings. Preparing to unpack .../041-fonts-glyphicons-halflings_1.009~3.4.1+dfsg-3_all.deb ... Unpacking fonts-glyphicons-halflings (1.009~3.4.1+dfsg-3) ... Selecting previously unselected package libjs-bootstrap. Preparing to unpack .../042-libjs-bootstrap_3.4.1+dfsg-3_all.deb ... Unpacking libjs-bootstrap (3.4.1+dfsg-3) ... Selecting previously unselected package python3-schema-salad. Preparing to unpack .../043-python3-schema-salad_8.5.20240311110950-1_armhf.deb ... Unpacking python3-schema-salad (8.5.20240311110950-1) ... Selecting previously unselected package cwl-upgrader. Preparing to unpack .../044-cwl-upgrader_1.2.11-1_all.deb ... Unpacking cwl-upgrader (1.2.11-1) ... Selecting previously unselected package python3-argcomplete. Preparing to unpack .../045-python3-argcomplete_3.3.0-1_all.deb ... Unpacking python3-argcomplete (3.3.0-1) ... Selecting previously unselected package python3-humanfriendly. Preparing to unpack .../046-python3-humanfriendly_10.0-5_all.deb ... Unpacking python3-humanfriendly (10.0-5) ... Selecting previously unselected package python3-coloredlogs. Preparing to unpack .../047-python3-coloredlogs_15.0.1-1_all.deb ... Unpacking python3-coloredlogs (15.0.1-1) ... Selecting previously unselected package python3-packaging. Preparing to unpack .../048-python3-packaging_24.0-1_all.deb ... Unpacking python3-packaging (24.0-1) ... Selecting previously unselected package python3-cwl-utils. Preparing to unpack .../049-python3-cwl-utils_0.33-1_all.deb ... Unpacking python3-cwl-utils (0.33-1) ... Selecting previously unselected package python3-dateutil. Preparing to unpack .../050-python3-dateutil_2.9.0-2_all.deb ... Unpacking python3-dateutil (2.9.0-2) ... Selecting previously unselected package libicu72:armhf. Preparing to unpack .../051-libicu72_72.1-4+b1_armhf.deb ... Unpacking libicu72:armhf (72.1-4+b1) ... Selecting previously unselected package libxml2:armhf. Preparing to unpack .../052-libxml2_2.9.14+dfsg-1.3+b3_armhf.deb ... Unpacking libxml2:armhf (2.9.14+dfsg-1.3+b3) ... Selecting previously unselected package libxslt1.1:armhf. Preparing to unpack .../053-libxslt1.1_1.1.35-1+b1_armhf.deb ... Unpacking libxslt1.1:armhf (1.1.35-1+b1) ... Selecting previously unselected package python3-lxml:armhf. Preparing to unpack .../054-python3-lxml_5.2.1-1_armhf.deb ... Unpacking python3-lxml:armhf (5.2.1-1) ... Selecting previously unselected package python3-networkx. Preparing to unpack .../055-python3-networkx_3.2.1-3_all.deb ... Unpacking python3-networkx (3.2.1-3) ... Selecting previously unselected package python3-prov. Preparing to unpack .../056-python3-prov_2.0.0-2_all.deb ... Unpacking python3-prov (2.0.0-2) ... Selecting previously unselected package python3-psutil. Preparing to unpack .../057-python3-psutil_5.9.8-2_armhf.deb ... Unpacking python3-psutil (5.9.8-2) ... Selecting previously unselected package libcdt5:armhf. Preparing to unpack .../058-libcdt5_2.42.2-9+b1_armhf.deb ... Unpacking libcdt5:armhf (2.42.2-9+b1) ... Selecting previously unselected package libcgraph6:armhf. Preparing to unpack .../059-libcgraph6_2.42.2-9+b1_armhf.deb ... Unpacking libcgraph6:armhf (2.42.2-9+b1) ... Selecting previously unselected package libaom3:armhf. Preparing to unpack .../060-libaom3_3.8.2-2_armhf.deb ... Unpacking libaom3:armhf (3.8.2-2) ... Selecting previously unselected package libdav1d7:armhf. Preparing to unpack .../061-libdav1d7_1.4.1-1_armhf.deb ... Unpacking libdav1d7:armhf (1.4.1-1) ... Selecting previously unselected package libabsl20230802:armhf. Preparing to unpack .../062-libabsl20230802_20230802.1-4_armhf.deb ... Unpacking libabsl20230802:armhf (20230802.1-4) ... Selecting previously unselected package libgav1-1:armhf. Preparing to unpack .../063-libgav1-1_0.19.0-2+b1_armhf.deb ... Unpacking libgav1-1:armhf (0.19.0-2+b1) ... Selecting previously unselected package librav1e0:armhf. Preparing to unpack .../064-librav1e0_0.7.1-2_armhf.deb ... Unpacking librav1e0:armhf (0.7.1-2) ... Selecting previously unselected package libsharpyuv0:armhf. Preparing to unpack .../065-libsharpyuv0_1.3.2-0.4+b1_armhf.deb ... Unpacking libsharpyuv0:armhf (1.3.2-0.4+b1) ... Selecting previously unselected package libsvtav1enc1d1:armhf. Preparing to unpack .../066-libsvtav1enc1d1_1.7.0+dfsg-2+b1_armhf.deb ... Unpacking libsvtav1enc1d1:armhf (1.7.0+dfsg-2+b1) ... Selecting previously unselected package libjpeg62-turbo:armhf. Preparing to unpack .../067-libjpeg62-turbo_1%3a2.1.5-3_armhf.deb ... Unpacking libjpeg62-turbo:armhf (1:2.1.5-3) ... Selecting previously unselected package libyuv0:armhf. Preparing to unpack .../068-libyuv0_0.0~git202401110.af6ac82-1_armhf.deb ... Unpacking libyuv0:armhf (0.0~git202401110.af6ac82-1) ... Selecting previously unselected package libavif16:armhf. Preparing to unpack .../069-libavif16_1.0.4-2_armhf.deb ... Unpacking libavif16:armhf (1.0.4-2) ... Selecting previously unselected package libbrotli1:armhf. Preparing to unpack .../070-libbrotli1_1.1.0-2+b3_armhf.deb ... Unpacking libbrotli1:armhf (1.1.0-2+b3) ... Selecting previously unselected package libpng16-16t64:armhf. Preparing to unpack .../071-libpng16-16t64_1.6.43-5_armhf.deb ... Unpacking libpng16-16t64:armhf (1.6.43-5) ... Selecting previously unselected package libfreetype6:armhf. Preparing to unpack .../072-libfreetype6_2.13.2+dfsg-1+b4_armhf.deb ... Unpacking libfreetype6:armhf (2.13.2+dfsg-1+b4) ... Selecting previously unselected package fonts-dejavu-mono. Preparing to unpack .../073-fonts-dejavu-mono_2.37-8_all.deb ... Unpacking fonts-dejavu-mono (2.37-8) ... Selecting previously unselected package fonts-dejavu-core. Preparing to unpack .../074-fonts-dejavu-core_2.37-8_all.deb ... Unpacking fonts-dejavu-core (2.37-8) ... Selecting previously unselected package fontconfig-config. Preparing to unpack .../075-fontconfig-config_2.15.0-1.1_armhf.deb ... Unpacking fontconfig-config (2.15.0-1.1) ... Selecting previously unselected package libfontconfig1:armhf. Preparing to unpack .../076-libfontconfig1_2.15.0-1.1_armhf.deb ... Unpacking libfontconfig1:armhf (2.15.0-1.1) ... Selecting previously unselected package libheif-plugin-dav1d:armhf. Preparing to unpack .../077-libheif-plugin-dav1d_1.17.6-1+b2_armhf.deb ... Unpacking libheif-plugin-dav1d:armhf (1.17.6-1+b2) ... Selecting previously unselected package libde265-0:armhf. Preparing to unpack .../078-libde265-0_1.0.15-1+b1_armhf.deb ... Unpacking libde265-0:armhf (1.0.15-1+b1) ... Selecting previously unselected package libheif-plugin-libde265:armhf. Preparing to unpack .../079-libheif-plugin-libde265_1.17.6-1+b2_armhf.deb ... Unpacking libheif-plugin-libde265:armhf (1.17.6-1+b2) ... Selecting previously unselected package libheif1:armhf. Preparing to unpack .../080-libheif1_1.17.6-1+b2_armhf.deb ... Unpacking libheif1:armhf (1.17.6-1+b2) ... Selecting previously unselected package libdeflate0:armhf. Preparing to unpack .../081-libdeflate0_1.20-1_armhf.deb ... Unpacking libdeflate0:armhf (1.20-1) ... Selecting previously unselected package libjbig0:armhf. Preparing to unpack .../082-libjbig0_2.1-6.1+b1_armhf.deb ... Unpacking libjbig0:armhf (2.1-6.1+b1) ... Selecting previously unselected package liblerc4:armhf. Preparing to unpack .../083-liblerc4_4.0.0+ds-4+b1_armhf.deb ... Unpacking liblerc4:armhf (4.0.0+ds-4+b1) ... Selecting previously unselected package libwebp7:armhf. Preparing to unpack .../084-libwebp7_1.3.2-0.4+b1_armhf.deb ... Unpacking libwebp7:armhf (1.3.2-0.4+b1) ... Selecting previously unselected package libtiff6:armhf. Preparing to unpack .../085-libtiff6_4.5.1+git230720-4_armhf.deb ... Unpacking libtiff6:armhf (4.5.1+git230720-4) ... Selecting previously unselected package libxau6:armhf. Preparing to unpack .../086-libxau6_1%3a1.0.9-1+b1_armhf.deb ... Unpacking libxau6:armhf (1:1.0.9-1+b1) ... Selecting previously unselected package libbsd0:armhf. Preparing to unpack .../087-libbsd0_0.12.2-1_armhf.deb ... Unpacking libbsd0:armhf (0.12.2-1) ... Selecting previously unselected package libxdmcp6:armhf. Preparing to unpack .../088-libxdmcp6_1%3a1.1.2-3+b1_armhf.deb ... Unpacking libxdmcp6:armhf (1:1.1.2-3+b1) ... Selecting previously unselected package libxcb1:armhf. Preparing to unpack .../089-libxcb1_1.17.0-1_armhf.deb ... Unpacking libxcb1:armhf (1.17.0-1) ... Selecting previously unselected package libx11-data. Preparing to unpack .../090-libx11-data_2%3a1.8.7-1_all.deb ... Unpacking libx11-data (2:1.8.7-1) ... Selecting previously unselected package libx11-6:armhf. Preparing to unpack .../091-libx11-6_2%3a1.8.7-1+b1_armhf.deb ... Unpacking libx11-6:armhf (2:1.8.7-1+b1) ... Selecting previously unselected package libxpm4:armhf. Preparing to unpack .../092-libxpm4_1%3a3.5.17-1+b1_armhf.deb ... Unpacking libxpm4:armhf (1:3.5.17-1+b1) ... Selecting previously unselected package libgd3:armhf. Preparing to unpack .../093-libgd3_2.3.3-9+b3_armhf.deb ... Unpacking libgd3:armhf (2.3.3-9+b3) ... Selecting previously unselected package libglib2.0-0t64:armhf. Preparing to unpack .../094-libglib2.0-0t64_2.78.4-7_armhf.deb ... Unpacking libglib2.0-0t64:armhf (2.78.4-7) ... Selecting previously unselected package libgts-0.7-5t64:armhf. Preparing to unpack .../095-libgts-0.7-5t64_0.7.6+darcs121130-5.2_armhf.deb ... Unpacking libgts-0.7-5t64:armhf (0.7.6+darcs121130-5.2) ... Selecting previously unselected package libpixman-1-0:armhf. Preparing to unpack .../096-libpixman-1-0_0.42.2-1+b1_armhf.deb ... Unpacking libpixman-1-0:armhf (0.42.2-1+b1) ... Selecting previously unselected package libxcb-render0:armhf. Preparing to unpack .../097-libxcb-render0_1.17.0-1_armhf.deb ... Unpacking libxcb-render0:armhf (1.17.0-1) ... Selecting previously unselected package libxcb-shm0:armhf. Preparing to unpack .../098-libxcb-shm0_1.17.0-1_armhf.deb ... Unpacking libxcb-shm0:armhf (1.17.0-1) ... Selecting previously unselected package libxext6:armhf. Preparing to unpack .../099-libxext6_2%3a1.3.4-1+b1_armhf.deb ... Unpacking libxext6:armhf (2:1.3.4-1+b1) ... Selecting previously unselected package libxrender1:armhf. Preparing to unpack .../100-libxrender1_1%3a0.9.10-1.1+b1_armhf.deb ... Unpacking libxrender1:armhf (1:0.9.10-1.1+b1) ... Selecting previously unselected package libcairo2:armhf. Preparing to unpack .../101-libcairo2_1.18.0-3+b1_armhf.deb ... Unpacking libcairo2:armhf (1.18.0-3+b1) ... Selecting previously unselected package libltdl7:armhf. Preparing to unpack .../102-libltdl7_2.4.7-7+b1_armhf.deb ... Unpacking libltdl7:armhf (2.4.7-7+b1) ... Selecting previously unselected package fontconfig. Preparing to unpack .../103-fontconfig_2.15.0-1.1_armhf.deb ... Unpacking fontconfig (2.15.0-1.1) ... Selecting previously unselected package libfribidi0:armhf. Preparing to unpack .../104-libfribidi0_1.0.13-3+b1_armhf.deb ... Unpacking libfribidi0:armhf (1.0.13-3+b1) ... Selecting previously unselected package libgraphite2-3:armhf. Preparing to unpack .../105-libgraphite2-3_1.3.14-2_armhf.deb ... Unpacking libgraphite2-3:armhf (1.3.14-2) ... Selecting previously unselected package libharfbuzz0b:armhf. Preparing to unpack .../106-libharfbuzz0b_8.3.0-2+b1_armhf.deb ... Unpacking libharfbuzz0b:armhf (8.3.0-2+b1) ... Selecting previously unselected package libthai-data. Preparing to unpack .../107-libthai-data_0.1.29-2_all.deb ... Unpacking libthai-data (0.1.29-2) ... Selecting previously unselected package libdatrie1:armhf. Preparing to unpack .../108-libdatrie1_0.2.13-3_armhf.deb ... Unpacking libdatrie1:armhf (0.2.13-3) ... Selecting previously unselected package libthai0:armhf. Preparing to unpack .../109-libthai0_0.1.29-2_armhf.deb ... Unpacking libthai0:armhf (0.1.29-2) ... Selecting previously unselected package libpango-1.0-0:armhf. Preparing to unpack .../110-libpango-1.0-0_1.52.1+ds-1_armhf.deb ... Unpacking libpango-1.0-0:armhf (1.52.1+ds-1) ... Selecting previously unselected package libpangoft2-1.0-0:armhf. Preparing to unpack .../111-libpangoft2-1.0-0_1.52.1+ds-1_armhf.deb ... Unpacking libpangoft2-1.0-0:armhf (1.52.1+ds-1) ... Selecting previously unselected package libpangocairo-1.0-0:armhf. Preparing to unpack .../112-libpangocairo-1.0-0_1.52.1+ds-1_armhf.deb ... Unpacking libpangocairo-1.0-0:armhf (1.52.1+ds-1) ... Selecting previously unselected package libpathplan4:armhf. Preparing to unpack .../113-libpathplan4_2.42.2-9+b1_armhf.deb ... Unpacking libpathplan4:armhf (2.42.2-9+b1) ... Selecting previously unselected package libgvc6. Preparing to unpack .../114-libgvc6_2.42.2-9+b1_armhf.deb ... Unpacking libgvc6 (2.42.2-9+b1) ... Selecting previously unselected package libgvpr2:armhf. Preparing to unpack .../115-libgvpr2_2.42.2-9+b1_armhf.deb ... Unpacking libgvpr2:armhf (2.42.2-9+b1) ... Selecting previously unselected package liblab-gamut1:armhf. Preparing to unpack .../116-liblab-gamut1_2.42.2-9+b1_armhf.deb ... Unpacking liblab-gamut1:armhf (2.42.2-9+b1) ... Selecting previously unselected package x11-common. Preparing to unpack .../117-x11-common_1%3a7.7+23_all.deb ... Unpacking x11-common (1:7.7+23) ... Selecting previously unselected package libice6:armhf. Preparing to unpack .../118-libice6_2%3a1.0.10-1+b1_armhf.deb ... Unpacking libice6:armhf (2:1.0.10-1+b1) ... Selecting previously unselected package libsm6:armhf. Preparing to unpack .../119-libsm6_2%3a1.2.3-1+b1_armhf.deb ... Unpacking libsm6:armhf (2:1.2.3-1+b1) ... Selecting previously unselected package libxt6t64:armhf. Preparing to unpack .../120-libxt6t64_1%3a1.2.1-1.2_armhf.deb ... Unpacking libxt6t64:armhf (1:1.2.1-1.2) ... Selecting previously unselected package libxmu6:armhf. Preparing to unpack .../121-libxmu6_2%3a1.1.3-3+b2_armhf.deb ... Unpacking libxmu6:armhf (2:1.1.3-3+b2) ... Selecting previously unselected package libxaw7:armhf. Preparing to unpack .../122-libxaw7_2%3a1.0.14-1+b2_armhf.deb ... Unpacking libxaw7:armhf (2:1.0.14-1+b2) ... Selecting previously unselected package graphviz. Preparing to unpack .../123-graphviz_2.42.2-9+b1_armhf.deb ... Unpacking graphviz (2.42.2-9+b1) ... Selecting previously unselected package python3-pydot. Preparing to unpack .../124-python3-pydot_2.0.0-2_all.deb ... Unpacking python3-pydot (2.0.0-2) ... Selecting previously unselected package python3-shellescape. Preparing to unpack .../125-python3-shellescape_3.4.1-5_all.deb ... Unpacking python3-shellescape (3.4.1-5) ... Selecting previously unselected package python3-spython. Preparing to unpack .../126-python3-spython_0.3.13-2_all.deb ... Unpacking python3-spython (0.3.13-2) ... Selecting previously unselected package python3-arcp. Preparing to unpack .../127-python3-arcp_0.2.1-6_all.deb ... Unpacking python3-arcp (0.2.1-6) ... Selecting previously unselected package python3-lib2to3. Preparing to unpack .../128-python3-lib2to3_3.12.3-1_all.deb ... Unpacking python3-lib2to3 (3.12.3-1) ... Selecting previously unselected package python3-distutils. Preparing to unpack .../129-python3-distutils_3.12.3-1_all.deb ... Unpacking python3-distutils (3.12.3-1) ... Selecting previously unselected package python3-setuptools. Preparing to unpack .../130-python3-setuptools_68.1.2-2_all.deb ... Unpacking python3-setuptools (68.1.2-2) ... Selecting previously unselected package cwltool. Preparing to unpack .../131-cwltool_3.1.20240404144621-2_all.deb ... Unpacking cwltool (3.1.20240404144621-2) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../132-libdebhelper-perl_13.15.3_all.deb ... Unpacking libdebhelper-perl (13.15.3) ... Selecting previously unselected package libtool. Preparing to unpack .../133-libtool_2.4.7-7_all.deb ... Unpacking libtool (2.4.7-7) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../134-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../135-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libsub-override-perl. Preparing to unpack .../136-libsub-override-perl_0.10-1_all.deb ... Unpacking libsub-override-perl (0.10-1) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../137-libfile-stripnondeterminism-perl_1.13.1-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.13.1-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../138-dh-strip-nondeterminism_1.13.1-1_all.deb ... Unpacking dh-strip-nondeterminism (1.13.1-1) ... Selecting previously unselected package libelf1t64:armhf. Preparing to unpack .../139-libelf1t64_0.191-1+b1_armhf.deb ... Unpacking libelf1t64:armhf (0.191-1+b1) ... Selecting previously unselected package dwz. Preparing to unpack .../140-dwz_0.15-1+b2_armhf.deb ... Unpacking dwz (0.15-1+b2) ... Selecting previously unselected package gettext. Preparing to unpack .../141-gettext_0.21-14+b1_armhf.deb ... Unpacking gettext (0.21-14+b1) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../142-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../143-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../144-debhelper_13.15.3_all.deb ... Unpacking debhelper (13.15.3) ... Selecting previously unselected package dh-python. Preparing to unpack .../145-dh-python_6.20240422_all.deb ... Unpacking dh-python (6.20240422) ... Selecting previously unselected package xml-core. Preparing to unpack .../146-xml-core_0.19_all.deb ... Unpacking xml-core (0.19) ... Selecting previously unselected package docutils-common. Preparing to unpack .../147-docutils-common_0.20.1+dfsg-3_all.deb ... Unpacking docutils-common (0.20.1+dfsg-3) ... Selecting previously unselected package libjs-jquery. Preparing to unpack .../148-libjs-jquery_3.6.1+dfsg+~3.5.14-1_all.deb ... Unpacking libjs-jquery (3.6.1+dfsg+~3.5.14-1) ... Selecting previously unselected package libjs-underscore. Preparing to unpack .../149-libjs-underscore_1.13.4~dfsg+~1.11.4-3_all.deb ... Unpacking libjs-underscore (1.13.4~dfsg+~1.11.4-3) ... Selecting previously unselected package libjs-sphinxdoc. Preparing to unpack .../150-libjs-sphinxdoc_7.2.6-6_all.deb ... Unpacking libjs-sphinxdoc (7.2.6-6) ... Selecting previously unselected package libjson-perl. Preparing to unpack .../151-libjson-perl_4.10000-1_all.deb ... Unpacking libjson-perl (4.10000-1) ... Selecting previously unselected package libpython3.12-stdlib:armhf. Preparing to unpack .../152-libpython3.12-stdlib_3.12.3-1_armhf.deb ... Unpacking libpython3.12-stdlib:armhf (3.12.3-1) ... Selecting previously unselected package libyaml-0-2:armhf. Preparing to unpack .../153-libyaml-0-2_0.2.5-1+b1_armhf.deb ... Unpacking libyaml-0-2:armhf (0.2.5-1+b1) ... Selecting previously unselected package python-babel-localedata. Preparing to unpack .../154-python-babel-localedata_2.14.0-1_all.deb ... Unpacking python-babel-localedata (2.14.0-1) ... Selecting previously unselected package python3-alabaster. Preparing to unpack .../155-python3-alabaster_0.7.12-1_all.deb ... Unpacking python3-alabaster (0.7.12-1) ... Selecting previously unselected package python3.12. Preparing to unpack .../156-python3.12_3.12.3-1_armhf.deb ... Unpacking python3.12 (3.12.3-1) ... Selecting previously unselected package python3-all. Preparing to unpack .../157-python3-all_3.11.8-1_armhf.deb ... Unpacking python3-all (3.11.8-1) ... Selecting previously unselected package python3-antlr4. Preparing to unpack .../158-python3-antlr4_4.9.2-1_all.deb ... Unpacking python3-antlr4 (4.9.2-1) ... Selecting previously unselected package python3-typing-extensions. Preparing to unpack .../159-python3-typing-extensions_4.10.0-1_all.deb ... Unpacking python3-typing-extensions (4.10.0-1) ... Selecting previously unselected package python3-astroid. Preparing to unpack .../160-python3-astroid_3.1.0-1_all.deb ... Unpacking python3-astroid (3.1.0-1) ... Selecting previously unselected package python3-tz. Preparing to unpack .../161-python3-tz_2024.1-2_all.deb ... Unpacking python3-tz (2024.1-2) ... Selecting previously unselected package python3-babel. Preparing to unpack .../162-python3-babel_2.14.0-1_all.deb ... Unpacking python3-babel (2.14.0-1) ... Selecting previously unselected package python3-wcwidth. Preparing to unpack .../163-python3-wcwidth_0.2.13+dfsg1-1_all.deb ... Unpacking python3-wcwidth (0.2.13+dfsg1-1) ... Selecting previously unselected package python3-blessed. Preparing to unpack .../164-python3-blessed_1.20.0-1_all.deb ... Unpacking python3-blessed (1.20.0-1) ... Selecting previously unselected package python3-jmespath. Preparing to unpack .../165-python3-jmespath_1.0.1-1_all.deb ... Unpacking python3-jmespath (1.0.1-1) ... Selecting previously unselected package python3-botocore. Preparing to unpack .../166-python3-botocore_1.34.46+repack-1_all.deb ... Unpacking python3-botocore (1.34.46+repack-1) ... Selecting previously unselected package python3-s3transfer. Preparing to unpack .../167-python3-s3transfer_0.10.1-1_all.deb ... Unpacking python3-s3transfer (0.10.1-1) ... Selecting previously unselected package python3-boto3. Preparing to unpack .../168-python3-boto3_1.34.46+dfsg-1_all.deb ... Unpacking python3-boto3 (1.34.46+dfsg-1) ... Selecting previously unselected package python3-configargparse. Preparing to unpack .../169-python3-configargparse_1.7-1_all.deb ... Unpacking python3-configargparse (1.7-1) ... Selecting previously unselected package python3-dill. Preparing to unpack .../170-python3-dill_0.3.8-1_all.deb ... Unpacking python3-dill (0.3.8-1) ... Selecting previously unselected package python3.11-doc. Preparing to unpack .../171-python3.11-doc_3.11.9-1_all.deb ... Unpacking python3.11-doc (3.11.9-1) ... Selecting previously unselected package python3-doc. Preparing to unpack .../172-python3-doc_3.11.8-1_all.deb ... Unpacking python3-doc (3.11.8-1) ... Selecting previously unselected package python3-websocket. Preparing to unpack .../173-python3-websocket_1.7.0-1_all.deb ... Unpacking python3-websocket (1.7.0-1) ... Selecting previously unselected package python3-docker. Preparing to unpack .../174-python3-docker_6.1.3-0.1_all.deb ... Unpacking python3-docker (6.1.3-0.1) ... Selecting previously unselected package python3-roman. Preparing to unpack .../175-python3-roman_3.3-3_all.deb ... Unpacking python3-roman (3.3-3) ... Selecting previously unselected package python3-docutils. Preparing to unpack .../176-python3-docutils_0.20.1+dfsg-3_all.deb ... Unpacking python3-docutils (0.20.1+dfsg-3) ... Selecting previously unselected package python3-prefixed. Preparing to unpack .../177-python3-prefixed_0.7.0-1_all.deb ... Unpacking python3-prefixed (0.7.0-1) ... Selecting previously unselected package python3-enlighten. Preparing to unpack .../178-python3-enlighten_1.11.2-1_armhf.deb ... Unpacking python3-enlighten (1.11.2-1) ... Selecting previously unselected package python3-execnet. Preparing to unpack .../179-python3-execnet_2.1.1-1_all.deb ... Unpacking python3-execnet (2.1.1-1) ... Selecting previously unselected package python3-imagesize. Preparing to unpack .../180-python3-imagesize_1.4.1-1_all.deb ... Unpacking python3-imagesize (1.4.1-1) ... Selecting previously unselected package python3-iniconfig. Preparing to unpack .../181-python3-iniconfig_1.1.1-2_all.deb ... Unpacking python3-iniconfig (1.1.1-2) ... Selecting previously unselected package python3-markupsafe. Preparing to unpack .../182-python3-markupsafe_2.1.5-1_armhf.deb ... Unpacking python3-markupsafe (2.1.5-1) ... Selecting previously unselected package python3-jinja2. Preparing to unpack .../183-python3-jinja2_3.1.3-1_all.deb ... Unpacking python3-jinja2 (3.1.3-1) ... Selecting previously unselected package python3-pluggy. Preparing to unpack .../184-python3-pluggy_1.5.0-1_all.deb ... Unpacking python3-pluggy (1.5.0-1) ... Selecting previously unselected package python3-pubsub. Preparing to unpack .../185-python3-pubsub_4.0.3-7_all.deb ... Unpacking python3-pubsub (4.0.3-7) ... Selecting previously unselected package python3-pygments. Preparing to unpack .../186-python3-pygments_2.17.2+dfsg-1_all.deb ... Unpacking python3-pygments (2.17.2+dfsg-1) ... Selecting previously unselected package python3-pytest. Preparing to unpack .../187-python3-pytest_8.1.2-1_all.deb ... Unpacking python3-pytest (8.1.2-1) ... Selecting previously unselected package python3-pytest-xdist. Preparing to unpack .../188-python3-pytest-xdist_3.5.0-1_all.deb ... Unpacking python3-pytest-xdist (3.5.0-1) ... Selecting previously unselected package python3-snowballstemmer. Preparing to unpack .../189-python3-snowballstemmer_2.2.0-4_all.deb ... Unpacking python3-snowballstemmer (2.2.0-4) ... Selecting previously unselected package sphinx-common. Preparing to unpack .../190-sphinx-common_7.2.6-6_all.deb ... Unpacking sphinx-common (7.2.6-6) ... Selecting previously unselected package python3-sphinx. Preparing to unpack .../191-python3-sphinx_7.2.6-6_all.deb ... Unpacking python3-sphinx (7.2.6-6) ... Selecting previously unselected package python3-unidecode. Preparing to unpack .../192-python3-unidecode_1.3.8-1_all.deb ... Unpacking python3-unidecode (1.3.8-1) ... Selecting previously unselected package python3-yaml. Preparing to unpack .../193-python3-yaml_6.0.1-2_armhf.deb ... Unpacking python3-yaml (6.0.1-2) ... Selecting previously unselected package python3-sphinx-autoapi. Preparing to unpack .../194-python3-sphinx-autoapi_3.0.0-0.1_all.deb ... Unpacking python3-sphinx-autoapi (3.0.0-0.1) ... Selecting previously unselected package python3-sphinx-autodoc-typehints. Preparing to unpack .../195-python3-sphinx-autodoc-typehints_1.25.3-1_all.deb ... Unpacking python3-sphinx-autodoc-typehints (1.25.3-1) ... Selecting previously unselected package python3-sphinxcontrib.autoprogram. Preparing to unpack .../196-python3-sphinxcontrib.autoprogram_0.1.8-2_all.deb ... Unpacking python3-sphinxcontrib.autoprogram (0.1.8-2) ... Selecting previously unselected package python3-stubserver. Preparing to unpack .../197-python3-stubserver_1.1-4_all.deb ... Unpacking python3-stubserver (1.1-4) ... Selecting previously unselected package python3-wdlparse. Preparing to unpack .../198-python3-wdlparse_0.1.0-3_all.deb ... Unpacking python3-wdlparse (0.1.0-3) ... Setting up libip4tc2:armhf (1.8.10-3) ... Setting up media-types (10.1.0) ... Setting up libpipeline1:armhf (1.5.7-2) ... Setting up libgraphite2-3:armhf (1.3.14-2) ... Setting up libpixman-1-0:armhf (0.42.2-1+b1) ... Setting up libsharpyuv0:armhf (1.3.2-0.4+b1) ... Setting up libaom3:armhf (3.8.2-2) ... Setting up libxau6:armhf (1:1.0.9-1+b1) ... Setting up libicu72:armhf (72.1-4+b1) ... Setting up liblerc4:armhf (4.0.0+ds-4+b1) ... Setting up bsdextrautils (2.40-8) ... Setting up libdatrie1:armhf (0.2.13-3) ... Setting up libmagic-mgc (1:5.45-3) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up libyaml-0-2:armhf (0.2.5-1+b1) ... Setting up libip6tc2:armhf (1.8.10-3) ... Setting up fonts-glyphicons-halflings (1.009~3.4.1+dfsg-3) ... Setting up libdebhelper-perl (13.15.3) ... Setting up libbrotli1:armhf (1.1.0-2+b3) ... Setting up liblab-gamut1:armhf (2.42.2-9+b1) ... Setting up libmagic1t64:armhf (1:5.45-3) ... Setting up libpython3.12-minimal:armhf (3.12.3-1) ... Setting up x11-common (1:7.7+23) ... invoke-rc.d: could not determine current runlevel Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix. Setting up libdeflate0:armhf (1.20-1) ... Setting up gettext-base (0.21-14+b1) ... Setting up m4 (1.4.19-4) ... Setting up libsvtav1enc1d1:armhf (1.7.0+dfsg-2+b1) ... Setting up file (1:5.45-3) ... Setting up libjbig0:armhf (2.1-6.1+b1) ... Setting up librav1e0:armhf (0.7.1-2) ... Setting up runc (1.1.12+ds1-2) ... Setting up libelf1t64:armhf (0.191-1+b1) ... Setting up python-babel-localedata (2.14.0-1) ... Setting up tzdata (2024a-3) ... Current default time zone: 'Etc/UTC' Local time is now: Tue Apr 30 23:41:56 UTC 2024. Universal Time is now: Tue Apr 30 23:41:56 UTC 2024. Run 'dpkg-reconfigure tzdata' if you wish to change it. Setting up autotools-dev (20220109.1) ... Setting up libglib2.0-0t64:armhf (2.78.4-7) ... No schema files found: doing nothing. Setting up libjpeg62-turbo:armhf (1:2.1.5-3) ... Setting up libx11-data (2:1.8.7-1) ... Setting up libpathplan4:armhf (2.42.2-9+b1) ... Setting up libfribidi0:armhf (1.0.13-3+b1) ... Setting up fonts-dejavu-mono (2.37-8) ... Setting up libpng16-16t64:armhf (1.6.43-5) ... Setting up libmnl0:armhf (1.0.5-2+b1) ... Setting up tini (0.19.0-1) ... Setting up autopoint (0.21-14) ... Setting up fonts-dejavu-core (2.37-8) ... Setting up libxtables12:armhf (1.8.10-3) ... Setting up libltdl7:armhf (2.4.7-7+b1) ... Setting up autoconf (2.71-3) ... Setting up libwebp7:armhf (1.3.2-0.4+b1) ... Setting up dwz (0.15-1+b2) ... Setting up libdav1d7:armhf (1.4.1-1) ... Setting up sensible-utils (0.0.22) ... Setting up libtiff6:armhf (4.5.1+git230720-4) ... Setting up libnfnetlink0:armhf (1.0.2-3) ... Setting up libuchardet0:armhf (0.0.8-1+b1) ... Setting up libjson-perl (4.10000-1) ... Setting up libsub-override-perl (0.10-1) ... Setting up libthai-data (0.1.29-2) ... Setting up netbase (6.4) ... Setting up libabsl20230802:armhf (20230802.1-4) ... Setting up libgts-0.7-5t64:armhf (0.7.6+darcs121130-5.2) ... Setting up sgml-base (1.31) ... Setting up libcdt5:armhf (2.42.2-9+b1) ... Setting up libcgraph6:armhf (2.42.2-9+b1) ... Setting up libjs-jquery (3.6.1+dfsg+~3.5.14-1) ... Setting up libde265-0:armhf (1.0.15-1+b1) ... Setting up containerd (1.6.24~ds1-1) ... Setting up openssl (3.2.1-3) ... Setting up libbsd0:armhf (0.12.2-1) ... Setting up libyuv0:armhf (0.0~git202401110.af6ac82-1) ... Setting up readline-common (8.2-4) ... Setting up libxml2:armhf (2.9.14+dfsg-1.3+b3) ... Setting up libjs-bootstrap (3.4.1+dfsg-3) ... Setting up libjs-underscore (1.13.4~dfsg+~1.11.4-3) ... Setting up libpopt0:armhf (1.19+dfsg-1+b1) ... Setting up automake (1:1.16.5-1.3) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up libfile-stripnondeterminism-perl (1.13.1-1) ... Setting up python3.12-minimal (3.12.3-1) ... Setting up libice6:armhf (2:1.0.10-1+b1) ... Setting up libxdmcp6:armhf (1:1.1.2-3+b1) ... Setting up libxcb1:armhf (1.17.0-1) ... Setting up gettext (0.21-14+b1) ... Setting up libtool (2.4.7-7) ... Setting up libxcb-render0:armhf (1.17.0-1) ... Setting up fontconfig-config (2.15.0-1.1) ... Setting up libnftnl11:armhf (1.2.6-2+b1) ... Setting up libxcb-shm0:armhf (1.17.0-1) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libthai0:armhf (0.1.29-2) ... Setting up ca-certificates (20240203) ... Updating certificates in /etc/ssl/certs... 146 added, 0 removed; done. Setting up libfreetype6:armhf (2.13.2+dfsg-1+b4) ... Setting up python3.11-doc (3.11.9-1) ... Setting up libgav1-1:armhf (0.19.0-2+b1) ... Setting up libjs-sphinxdoc (7.2.6-6) ... Setting up libreadline8t64:armhf (8.2-4) ... Setting up dh-strip-nondeterminism (1.13.1-1) ... Setting up libgvpr2:armhf (2.42.2-9+b1) ... Setting up libnetfilter-conntrack3:armhf (1.0.9-6+b1) ... Setting up groff-base (1.23.0-4) ... Setting up xml-core (0.19) ... Setting up libxslt1.1:armhf (1.1.35-1+b1) ... Setting up libx11-6:armhf (2:1.8.7-1+b1) ... Setting up libharfbuzz0b:armhf (8.3.0-2+b1) ... Setting up libfontconfig1:armhf (2.15.0-1.1) ... Setting up libsm6:armhf (2:1.2.3-1+b1) ... Setting up python3-doc (3.11.8-1) ... Setting up rsync (3.3.0-1) ... invoke-rc.d: could not determine current runlevel invoke-rc.d: policy-rc.d denied execution of start. Setting up fontconfig (2.15.0-1.1) ... Regenerating fonts cache... done. Setting up libavif16:armhf (1.0.4-2) ... Setting up libpython3.12-stdlib:armhf (3.12.3-1) ... Setting up libxpm4:armhf (1:3.5.17-1+b1) ... Setting up libxrender1:armhf (1:0.9.10-1.1+b1) ... Setting up iptables (1.8.10-3) ... update-alternatives: using /usr/sbin/iptables-legacy to provide /usr/sbin/iptables (iptables) in auto mode update-alternatives: using /usr/sbin/ip6tables-legacy to provide /usr/sbin/ip6tables (ip6tables) in auto mode update-alternatives: using /usr/sbin/iptables-nft to provide /usr/sbin/iptables (iptables) in auto mode update-alternatives: using /usr/sbin/ip6tables-nft to provide /usr/sbin/ip6tables (ip6tables) in auto mode update-alternatives: using /usr/sbin/arptables-nft to provide /usr/sbin/arptables (arptables) in auto mode update-alternatives: using /usr/sbin/ebtables-nft to provide /usr/sbin/ebtables (ebtables) in auto mode Setting up po-debconf (1.0.21+nmu1) ... Setting up libpython3.11-stdlib:armhf (3.11.9-1) ... Setting up python3.12 (3.12.3-1) ... Setting up libpango-1.0-0:armhf (1.52.1+ds-1) ... Setting up libxext6:armhf (2:1.3.4-1+b1) ... Setting up man-db (2.12.1-1) ... Not building database; man-db/auto-update is not 'true'. Setting up libcairo2:armhf (1.18.0-3+b1) ... Setting up sphinx-common (7.2.6-6) ... Setting up libxt6t64:armhf (1:1.2.1-1.2) ... Setting up libpangoft2-1.0-0:armhf (1.52.1+ds-1) ... Setting up libpangocairo-1.0-0:armhf (1.52.1+ds-1) ... Setting up libpython3-stdlib:armhf (3.11.8-1) ... Setting up python3.11 (3.11.9-1) ... Setting up libxmu6:armhf (2:1.1.3-3+b2) ... Setting up debhelper (13.15.3) ... Setting up python3 (3.11.8-1) ... Setting up libxaw7:armhf (2:1.0.14-1+b2) ... Setting up python3-markupsafe (2.1.5-1) ... Setting up python3-psutil (5.9.8-2) ... Setting up python3-tz (2024.1-2) ... Setting up python3-prefixed (0.7.0-1) ... Setting up python3-stubserver (1.1-4) ... Setting up python3-six (1.16.0-6) ... Setting up python3-isodate (0.6.1-2) ... Setting up python3-roman (3.3-3) ... Setting up python3-jinja2 (3.1.3-1) ... Setting up python3-packaging (24.0-1) ... Setting up python3-wcwidth (0.2.13+dfsg1-1) ... Setting up python3-pyparsing (3.1.2-1) ... Setting up python3-humanfriendly (10.0-5) ... Setting up python3-certifi (2023.11.17-1) ... Setting up python3-spython (0.3.13-2) ... Setting up python3-snowballstemmer (2.2.0-4) ... Setting up python3-jmespath (1.0.1-1) ... Setting up python3-dill (0.3.8-1) ... Setting up python3-idna (3.6-2) ... Setting up python3-typing-extensions (4.10.0-1) ... Setting up python3-urllib3 (1.26.18-2) ... Setting up python3-pluggy (1.5.0-1) ... Setting up python3-pubsub (4.0.3-7) ... Setting up python3-lxml:armhf (5.2.1-1) ... Setting up python3-rdflib (6.3.2-1) ... Setting up python3-dateutil (2.9.0-2) ... Setting up python3-msgpack (1.0.3-3+b1) ... Setting up python3-shellescape (3.4.1-5) ... Setting up python3-argcomplete (3.3.0-1) ... Setting up python3-execnet (2.1.1-1) ... Setting up python3-lib2to3 (3.12.3-1) ... Setting up python3-coloredlogs (15.0.1-1) ... Setting up python3-mistune (3.0.2-1) ... Setting up python3-unidecode (1.3.8-1) ... Setting up python3-websocket (1.7.0-1) ... Setting up python3-imagesize (1.4.1-1) ... Setting up python3-pkg-resources (68.1.2-2) ... Setting up python3-distutils (3.12.3-1) ... python3.12: can't get files for byte-compilation Setting up python3-configargparse (1.7-1) ... Setting up python3-arcp (0.2.1-6) ... Setting up python3-iniconfig (1.1.1-2) ... Setting up python3-antlr4 (4.9.2-1) ... Setting up python3-filelock (3.13.4-1) ... Setting up python3-setuptools (68.1.2-2) ... Setting up python3-babel (2.14.0-1) ... update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode Setting up python3-charset-normalizer (3.3.2-1) ... Setting up python3-pytest (8.1.2-1) ... Setting up python3-alabaster (0.7.12-1) ... Setting up python3-mypy-extensions (1.0.0-1) ... Setting up python3-blessed (1.20.0-1) ... Setting up python3-ruamel.yaml.clib:armhf (0.2.8+ds-1) ... Setting up python3-all (3.11.8-1) ... Setting up python3-yaml (6.0.1-2) ... Setting up python3-networkx (3.2.1-3) ... /usr/lib/python3/dist-packages/networkx/readwrite/tests/test_gml.py:557: SyntaxWarning: invalid octal escape sequence '\420' "graph [edge [ source u'u\4200' target u'u\4200' ] " /usr/lib/python3/dist-packages/networkx/readwrite/tests/test_gml.py:558: SyntaxWarning: invalid octal escape sequence '\420' + "node [ id u'u\4200' label b ] ]" Setting up python3-prov (2.0.0-2) ... Setting up python3-enlighten (1.11.2-1) ... Setting up python3-pygments (2.17.2+dfsg-1) ... Setting up python3-chardet (5.2.0+dfsg-1) ... Setting up python3-requests (2.31.0+dfsg-1) ... Setting up python3-ruamel.yaml (0.18.6+ds-3) ... Setting up python3-astroid (3.1.0-1) ... Setting up python3-cachecontrol (0.14.0-1) ... Setting up python3-wdlparse (0.1.0-3) ... Setting up dh-python (6.20240422) ... Setting up python3-pytest-xdist (3.5.0-1) ... Setting up python3-docker (6.1.3-0.1) ... Setting up python3-schema-salad (8.5.20240311110950-1) ... Setting up cwl-upgrader (1.2.11-1) ... Setting up python3-botocore (1.34.46+repack-1) ... Setting up python3-s3transfer (0.10.1-1) ... Setting up python3-cwl-utils (0.33-1) ... Setting up python3-boto3 (1.34.46+dfsg-1) ... Setting up libheif-plugin-libde265:armhf (1.17.6-1+b2) ... Setting up libheif1:armhf (1.17.6-1+b2) ... Setting up libgd3:armhf (2.3.3-9+b3) ... Setting up libgvc6 (2.42.2-9+b1) ... Setting up libdevmapper1.02.1:armhf (2:1.02.196-1+b1) ... Setting up dmsetup (2:1.02.196-1+b1) ... Setting up libheif-plugin-dav1d:armhf (1.17.6-1+b2) ... Setting up graphviz (2.42.2-9+b1) ... Setting up python3-pydot (2.0.0-2) ... Setting up docker.io (20.10.25+dfsg1-2+b3) ... info: Selecting GID from range 100 to 999 ... info: Adding group `docker' (GID 101) ... invoke-rc.d: could not determine current runlevel invoke-rc.d: policy-rc.d denied execution of start. Setting up cwltool (3.1.20240404144621-2) ... Processing triggers for libc-bin (2.37-19) ... Processing triggers for sgml-base (1.31) ... Setting up docutils-common (0.20.1+dfsg-3) ... Processing triggers for sgml-base (1.31) ... Setting up python3-docutils (0.20.1+dfsg-3) ... Setting up python3-sphinx (7.2.6-6) ... Setting up python3-sphinx-autodoc-typehints (1.25.3-1) ... Setting up python3-sphinx-autoapi (3.0.0-0.1) ... Setting up python3-sphinxcontrib.autoprogram (0.1.8-2) ... Processing triggers for ca-certificates (20240203) ... Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps I: Building the package I: Running cd /build/reproducible-path/toil-6.1.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../toil_6.1.0-4_source.changes dpkg-buildpackage: info: source package toil dpkg-buildpackage: info: source version 6.1.0-4 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Michael R. Crusoe dpkg-source --before-build . dpkg-buildpackage: info: host architecture armhf dpkg-source: info: using options from toil-6.1.0/debian/source/options: --extend-diff-ignore=^[^/]*[.]egg-info/ debian/rules clean dh clean --buildsystem=pybuild debian/rules override_dh_auto_clean make[1]: Entering directory '/build/reproducible-path/toil-6.1.0' rm -rf .pybuild rm -rf docs/_build find . -name "*.pyc" -delete make[1]: Leaving directory '/build/reproducible-path/toil-6.1.0' dh_autoreconf_clean -O--buildsystem=pybuild dh_clean -O--buildsystem=pybuild debian/rules binary dh binary --buildsystem=pybuild dh_update_autotools_config -O--buildsystem=pybuild dh_autoreconf -O--buildsystem=pybuild dh_auto_configure -O--buildsystem=pybuild I: pybuild base:311: python3.12 setup.py config running config I: pybuild base:311: python3.11 setup.py config running config debian/rules override_dh_auto_build make[1]: Entering directory '/build/reproducible-path/toil-6.1.0' dh_auto_build I: pybuild base:311: /usr/bin/python3.12 setup.py build running build running build_py creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/statsAndLogging.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/version.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/worker.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/toilState.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/realtimeLogger.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/exceptions.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/serviceManager.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/leader.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/bus.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/common.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/deferred.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/resource.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil copying src/toil/job.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/options copying src/toil/options/cwl.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/options copying src/toil/options/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/options copying src/toil/options/common.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/options copying src/toil/options/wdl.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/options creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/torque.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/singleMachine.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/gridengine.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/kubernetes.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/options.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/lsf.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/abstractBatchSystem.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/contained_executor.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/registry.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/awsBatch.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/local_support.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/abstractGridEngineBatchSystem.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/lsfHelper.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/cleanup_support.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/slurm.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems copying src/toil/batchSystems/htcondor.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test copying src/toil/test/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/wdl copying src/toil/wdl/utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/wdl copying src/toil/wdl/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/wdl copying src/toil/wdl/wdltoil.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/wdl creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/cwl copying src/toil/cwl/utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/cwl copying src/toil/cwl/cwltoil.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/cwl copying src/toil/cwl/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/cwl copying src/toil/cwl/conftest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/cwl creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/jobStores copying src/toil/jobStores/utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/jobStores copying src/toil/jobStores/googleJobStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/jobStores copying src/toil/jobStores/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/jobStores copying src/toil/jobStores/fileJobStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/jobStores copying src/toil/jobStores/conftest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/jobStores copying src/toil/jobStores/abstractJobStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/jobStores creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/threading.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/generatedEC2Lists.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/humanize.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/throttle.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/compatibility.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/objects.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/memoize.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/ec2.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/accelerators.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/exceptions.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/expando.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/conversions.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/misc.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/resources.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/retry.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/ec2nodes.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/io.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/iterables.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/bioio.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib copying src/toil/lib/docker.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilConfig.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilSshCluster.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilDebugFile.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilKill.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilUpdateEC2Instances.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilDestroyCluster.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilServer.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilRsyncCluster.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilStats.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilDebugJob.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilStatus.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilClean.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilMain.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils copying src/toil/utils/toilLaunchCluster.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/utils creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/fileStores copying src/toil/fileStores/cachingFileStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/fileStores copying src/toil/fileStores/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/fileStores copying src/toil/fileStores/abstractFileStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/fileStores copying src/toil/fileStores/nonCachingFileStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/fileStores creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/provisioners copying src/toil/provisioners/gceProvisioner.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/provisioners copying src/toil/provisioners/node.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/provisioners copying src/toil/provisioners/abstractProvisioner.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/provisioners copying src/toil/provisioners/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/provisioners copying src/toil/provisioners/clusterScaler.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/provisioners creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server copying src/toil/server/utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server copying src/toil/server/app.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server copying src/toil/server/celery_app.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server copying src/toil/server/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server copying src/toil/server/wsgi_app.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems/mesos copying src/toil/batchSystems/mesos/executor.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems/mesos copying src/toil/batchSystems/mesos/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems/mesos copying src/toil/batchSystems/mesos/batchSystem.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems/mesos copying src/toil/batchSystems/mesos/conftest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems/mesos creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems/mesos/test copying src/toil/batchSystems/mesos/test/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/batchSystems/mesos/test creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/options copying src/toil/test/options/options.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/options copying src/toil/test/options/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/options creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/batchSystems copying src/toil/test/batchSystems/batchSystemTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/batchSystems copying src/toil/test/batchSystems/test_slurm.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/batchSystems copying src/toil/test/batchSystems/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/batchSystems copying src/toil/test/batchSystems/test_lsf_helper.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/batchSystems creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/docs copying src/toil/test/docs/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/docs copying src/toil/test/docs/scriptsTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/docs creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/mesos copying src/toil/test/mesos/MesosDataStructuresTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/mesos copying src/toil/test/mesos/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/mesos copying src/toil/test/mesos/stress.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/mesos copying src/toil/test/mesos/helloWorld.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/mesos creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/wdl copying src/toil/test/wdl/wdltoil_test.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/wdl copying src/toil/test/wdl/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/wdl creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/cwlTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/conftest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/userDefinedJobArgTypeTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/dockerCheckTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/helloWorldTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/retainTempDirTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/importExportFileTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/restartDAGTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/miscTests.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/autoDeploymentTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/jobServiceTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/promisesTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/systemTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/promisedRequirementTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/realtimeLoggerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/jobFileStoreTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/checkpointTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/resumabilityTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/jobEncapsulationTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/jobDescriptionTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/toilContextManagerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/regularLogTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/deferredFunctionTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/resourceTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/workerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/jobTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/threadingTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/fileStoreTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src copying src/toil/test/src/busTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/src creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/jobStores copying src/toil/test/jobStores/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/jobStores copying src/toil/test/jobStores/jobStoreTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/jobStores creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/lib copying src/toil/test/lib/test_ec2.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/lib copying src/toil/test/lib/test_conversions.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/lib copying src/toil/test/lib/dockerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/lib copying src/toil/test/lib/test_misc.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/lib copying src/toil/test/lib/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/lib creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/utils copying src/toil/test/utils/utilsTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/utils copying src/toil/test/utils/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/utils copying src/toil/test/utils/toilDebugTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/utils copying src/toil/test/utils/toilKillTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/utils creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/sort copying src/toil/test/sort/sort.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/sort copying src/toil/test/sort/sortTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/sort copying src/toil/test/sort/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/sort copying src/toil/test/sort/restart_sort.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/sort creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/provisioners copying src/toil/test/provisioners/restartScript.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/provisioners copying src/toil/test/provisioners/provisionerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/provisioners copying src/toil/test/provisioners/clusterScalerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/provisioners copying src/toil/test/provisioners/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/provisioners copying src/toil/test/provisioners/gceProvisionerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/provisioners copying src/toil/test/provisioners/clusterTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/provisioners creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/server copying src/toil/test/server/serverTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/server copying src/toil/test/server/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/server creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cactus copying src/toil/test/cactus/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cactus copying src/toil/test/cactus/test_cactus_integration.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cactus creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/lib/aws copying src/toil/test/lib/aws/test_s3.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/lib/aws copying src/toil/test/lib/aws/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/lib/aws copying src/toil/test/lib/aws/test_utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/lib/aws copying src/toil/test/lib/aws/test_iam.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/lib/aws creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/provisioners/aws copying src/toil/test/provisioners/aws/awsProvisionerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/provisioners/aws copying src/toil/test/provisioners/aws/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/provisioners/aws creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/jobStores/aws copying src/toil/jobStores/aws/utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/jobStores/aws copying src/toil/jobStores/aws/jobStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/jobStores/aws copying src/toil/jobStores/aws/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/jobStores/aws creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib/aws copying src/toil/lib/aws/utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib/aws copying src/toil/lib/aws/session.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib/aws copying src/toil/lib/aws/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib/aws copying src/toil/lib/aws/ami.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib/aws copying src/toil/lib/aws/iam.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib/aws creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib/encryption copying src/toil/lib/encryption/_nacl.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib/encryption copying src/toil/lib/encryption/_dummy.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib/encryption copying src/toil/lib/encryption/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib/encryption copying src/toil/lib/encryption/conftest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/lib/encryption creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/provisioners/aws copying src/toil/provisioners/aws/awsProvisioner.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/provisioners/aws copying src/toil/provisioners/aws/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/provisioners/aws creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server/wes copying src/toil/server/wes/abstract_backend.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server/wes copying src/toil/server/wes/toil_backend.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server/wes copying src/toil/server/wes/amazon_wes_utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server/wes copying src/toil/server/wes/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server/wes copying src/toil/server/wes/tasks.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server/wes creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server/cli copying src/toil/server/cli/wes_cwl_runner.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server/cli copying src/toil/server/cli/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server/cli creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server/api_spec copying src/toil/server/api_spec/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server/api_spec copying src/toil/test/cwl/colon_test_output_job.yaml -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/conditional_wf.yaml -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/directory_from_directory.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/stream.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/s3_secondary_file.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/alwaysfails.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/nvidia_smi.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/test_filename_conflict_detection.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/file_from_directory.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/echo-stdout-log-dir.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/preemptible.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/download.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/colon_test_output.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/preemptible_expression.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/glob_dir.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/load_contents.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/sorttool.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/conditional_wf.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/echo_string.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/test_filename_conflict_detection_at_root.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/revtool.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/revsort.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/test_filename_conflict_resolution.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/seqtk_seq.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/revsort2.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/echo_string_scatter_capture_stdout.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/download_subdirectory.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/download_directory.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/echo-stderr.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/mpi_simple.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/echo.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/test/cwl/revtool2.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/test/cwl copying src/toil/server/api_spec/workflow_execution_service.swagger.yaml -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/build/toil/server/api_spec I: pybuild base:311: /usr/bin/python3 setup.py build running build running build_py creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/statsAndLogging.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/version.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/worker.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/toilState.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/realtimeLogger.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/exceptions.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/serviceManager.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/leader.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/bus.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/common.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/deferred.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/resource.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil copying src/toil/job.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/options copying src/toil/options/cwl.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/options copying src/toil/options/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/options copying src/toil/options/common.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/options copying src/toil/options/wdl.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/options creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/torque.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/singleMachine.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/gridengine.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/kubernetes.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/options.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/lsf.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/abstractBatchSystem.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/contained_executor.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/registry.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/awsBatch.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/local_support.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/abstractGridEngineBatchSystem.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/lsfHelper.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/cleanup_support.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/slurm.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems copying src/toil/batchSystems/htcondor.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test copying src/toil/test/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/wdl copying src/toil/wdl/utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/wdl copying src/toil/wdl/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/wdl copying src/toil/wdl/wdltoil.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/wdl creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/cwl copying src/toil/cwl/utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/cwl copying src/toil/cwl/cwltoil.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/cwl copying src/toil/cwl/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/cwl copying src/toil/cwl/conftest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/cwl creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/jobStores copying src/toil/jobStores/utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/jobStores copying src/toil/jobStores/googleJobStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/jobStores copying src/toil/jobStores/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/jobStores copying src/toil/jobStores/fileJobStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/jobStores copying src/toil/jobStores/conftest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/jobStores copying src/toil/jobStores/abstractJobStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/jobStores creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/threading.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/generatedEC2Lists.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/humanize.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/throttle.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/compatibility.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/objects.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/memoize.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/ec2.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/accelerators.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/exceptions.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/expando.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/conversions.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/misc.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/resources.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/retry.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/ec2nodes.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/io.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/iterables.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/bioio.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib copying src/toil/lib/docker.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilConfig.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilSshCluster.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilDebugFile.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilKill.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilUpdateEC2Instances.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilDestroyCluster.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilServer.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilRsyncCluster.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilStats.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilDebugJob.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilStatus.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilClean.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilMain.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils copying src/toil/utils/toilLaunchCluster.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/utils creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/fileStores copying src/toil/fileStores/cachingFileStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/fileStores copying src/toil/fileStores/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/fileStores copying src/toil/fileStores/abstractFileStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/fileStores copying src/toil/fileStores/nonCachingFileStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/fileStores creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/provisioners copying src/toil/provisioners/gceProvisioner.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/provisioners copying src/toil/provisioners/node.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/provisioners copying src/toil/provisioners/abstractProvisioner.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/provisioners copying src/toil/provisioners/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/provisioners copying src/toil/provisioners/clusterScaler.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/provisioners creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server copying src/toil/server/utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server copying src/toil/server/app.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server copying src/toil/server/celery_app.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server copying src/toil/server/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server copying src/toil/server/wsgi_app.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems/mesos copying src/toil/batchSystems/mesos/executor.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems/mesos copying src/toil/batchSystems/mesos/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems/mesos copying src/toil/batchSystems/mesos/batchSystem.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems/mesos copying src/toil/batchSystems/mesos/conftest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems/mesos creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems/mesos/test copying src/toil/batchSystems/mesos/test/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/batchSystems/mesos/test creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/options copying src/toil/test/options/options.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/options copying src/toil/test/options/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/options creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/batchSystems copying src/toil/test/batchSystems/batchSystemTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/batchSystems copying src/toil/test/batchSystems/test_slurm.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/batchSystems copying src/toil/test/batchSystems/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/batchSystems copying src/toil/test/batchSystems/test_lsf_helper.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/batchSystems creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/docs copying src/toil/test/docs/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/docs copying src/toil/test/docs/scriptsTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/docs creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/mesos copying src/toil/test/mesos/MesosDataStructuresTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/mesos copying src/toil/test/mesos/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/mesos copying src/toil/test/mesos/stress.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/mesos copying src/toil/test/mesos/helloWorld.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/mesos creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/wdl copying src/toil/test/wdl/wdltoil_test.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/wdl copying src/toil/test/wdl/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/wdl creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/cwlTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/conftest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/userDefinedJobArgTypeTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/dockerCheckTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/helloWorldTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/retainTempDirTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/importExportFileTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/restartDAGTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/miscTests.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/autoDeploymentTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/jobServiceTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/promisesTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/systemTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/promisedRequirementTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/realtimeLoggerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/jobFileStoreTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/checkpointTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/resumabilityTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/jobEncapsulationTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/jobDescriptionTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/toilContextManagerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/regularLogTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/deferredFunctionTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/resourceTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/workerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/jobTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/threadingTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/fileStoreTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src copying src/toil/test/src/busTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/src creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/jobStores copying src/toil/test/jobStores/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/jobStores copying src/toil/test/jobStores/jobStoreTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/jobStores creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/lib copying src/toil/test/lib/test_ec2.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/lib copying src/toil/test/lib/test_conversions.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/lib copying src/toil/test/lib/dockerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/lib copying src/toil/test/lib/test_misc.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/lib copying src/toil/test/lib/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/lib creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/utils copying src/toil/test/utils/utilsTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/utils copying src/toil/test/utils/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/utils copying src/toil/test/utils/toilDebugTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/utils copying src/toil/test/utils/toilKillTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/utils creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/sort copying src/toil/test/sort/sort.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/sort copying src/toil/test/sort/sortTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/sort copying src/toil/test/sort/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/sort copying src/toil/test/sort/restart_sort.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/sort creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/provisioners copying src/toil/test/provisioners/restartScript.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/provisioners copying src/toil/test/provisioners/provisionerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/provisioners copying src/toil/test/provisioners/clusterScalerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/provisioners copying src/toil/test/provisioners/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/provisioners copying src/toil/test/provisioners/gceProvisionerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/provisioners copying src/toil/test/provisioners/clusterTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/provisioners creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/server copying src/toil/test/server/serverTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/server copying src/toil/test/server/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/server creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cactus copying src/toil/test/cactus/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cactus copying src/toil/test/cactus/test_cactus_integration.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cactus creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/lib/aws copying src/toil/test/lib/aws/test_s3.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/lib/aws copying src/toil/test/lib/aws/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/lib/aws copying src/toil/test/lib/aws/test_utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/lib/aws copying src/toil/test/lib/aws/test_iam.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/lib/aws creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/provisioners/aws copying src/toil/test/provisioners/aws/awsProvisionerTest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/provisioners/aws copying src/toil/test/provisioners/aws/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/provisioners/aws creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/jobStores/aws copying src/toil/jobStores/aws/utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/jobStores/aws copying src/toil/jobStores/aws/jobStore.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/jobStores/aws copying src/toil/jobStores/aws/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/jobStores/aws creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib/aws copying src/toil/lib/aws/utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib/aws copying src/toil/lib/aws/session.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib/aws copying src/toil/lib/aws/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib/aws copying src/toil/lib/aws/ami.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib/aws copying src/toil/lib/aws/iam.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib/aws creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib/encryption copying src/toil/lib/encryption/_nacl.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib/encryption copying src/toil/lib/encryption/_dummy.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib/encryption copying src/toil/lib/encryption/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib/encryption copying src/toil/lib/encryption/conftest.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/lib/encryption creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/provisioners/aws copying src/toil/provisioners/aws/awsProvisioner.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/provisioners/aws copying src/toil/provisioners/aws/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/provisioners/aws creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server/wes copying src/toil/server/wes/abstract_backend.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server/wes copying src/toil/server/wes/toil_backend.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server/wes copying src/toil/server/wes/amazon_wes_utils.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server/wes copying src/toil/server/wes/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server/wes copying src/toil/server/wes/tasks.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server/wes creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server/cli copying src/toil/server/cli/wes_cwl_runner.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server/cli copying src/toil/server/cli/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server/cli creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server/api_spec copying src/toil/server/api_spec/__init__.py -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server/api_spec copying src/toil/test/cwl/colon_test_output_job.yaml -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/conditional_wf.yaml -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/directory_from_directory.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/stream.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/s3_secondary_file.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/alwaysfails.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/nvidia_smi.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/test_filename_conflict_detection.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/file_from_directory.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/echo-stdout-log-dir.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/preemptible.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/download.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/colon_test_output.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/preemptible_expression.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/glob_dir.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/load_contents.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/sorttool.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/conditional_wf.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/echo_string.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/test_filename_conflict_detection_at_root.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/revtool.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/revsort.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/test_filename_conflict_resolution.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/seqtk_seq.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/revsort2.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/echo_string_scatter_capture_stdout.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/download_subdirectory.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/download_directory.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/echo-stderr.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/mpi_simple.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/echo.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/test/cwl/revtool2.cwl -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/test/cwl copying src/toil/server/api_spec/workflow_execution_service.swagger.yaml -> /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/build/toil/server/api_spec PYTHONPATH=/build/reproducible-path/toil-6.1.0/src /usr/bin/make -C docs man SPHINXOPTS="" make[2]: Entering directory '/build/reproducible-path/toil-6.1.0/docs' Running Sphinx v7.2.6 making output directory... done [autosummary] generating autosummary for: appendices/architecture.rst, appendices/aws_min_permissions.rst, appendices/deploy.rst, appendices/environment_vars.rst, contributing/checklists.rst, contributing/contributing.rst, cwl/introduction.rst, cwl/running.rst, gettingStarted/install.rst, gettingStarted/quickStart.rst, ..., running/cloud/kubernetes.rst, running/debugging.rst, running/hpcEnvironments.rst, running/introduction.rst, running/server/wes.rst, running/utils.rst, wdl/conformance.rst, wdl/developing.rst, wdl/introduction.rst, wdl/running.rst [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_quickstart.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_docker.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_multiplejobs2.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_cwlexample.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_stats.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/example_alwaysfail.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_requirements.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/example_cachingbenchmark.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_jobfunctions.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_multiplejobs.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_encapsulation2.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_arguments.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_promises.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_managing.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_invokeworkflow.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_staging.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_multiplejobs3.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_promises2.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_managing2.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_dynamic.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_encapsulation.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_helloworld.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_invokeworkflow2.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scripts/tutorial_services.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/cwl/mock_mpi/fake_mpi_run.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/utils/ABCWorkflowDebug/debugWorkflow.py [AutoAPI] Ignoring /build/reproducible-path/toil-6.1.0/src/toil/test/utils/ABCWorkflowDebug/mkFile.py [AutoAPI] Reading files... [ 0%] /build/reproducible-path/toil-6.1.0/src/toil/statsAndLogging.py [AutoAPI] Reading files... [ 1%] /build/reproducible-path/toil-6.1.0/src/toil/version.py [AutoAPI] Reading files... [ 1%] /build/reproducible-path/toil-6.1.0/src/toil/worker.py [AutoAPI] Reading files... [ 2%] /build/reproducible-path/toil-6.1.0/src/toil/toilState.py [AutoAPI] Reading files... [ 2%] /build/reproducible-path/toil-6.1.0/src/toil/realtimeLogger.py [AutoAPI] Reading files... [ 3%] /build/reproducible-path/toil-6.1.0/src/toil/exceptions.py [AutoAPI] Reading files... [ 3%] /build/reproducible-path/toil-6.1.0/src/toil/__init__.py [AutoAPI] Reading files... [ 4%] /build/reproducible-path/toil-6.1.0/src/toil/serviceManager.py [AutoAPI] Reading files... [ 4%] /build/reproducible-path/toil-6.1.0/src/toil/leader.py [AutoAPI] Reading files... [ 5%] /build/reproducible-path/toil-6.1.0/src/toil/bus.py [AutoAPI] Reading files... [ 5%] /build/reproducible-path/toil-6.1.0/src/toil/common.py [AutoAPI] Reading files... [ 6%] /build/reproducible-path/toil-6.1.0/src/toil/deferred.py [AutoAPI] Reading files... [ 6%] /build/reproducible-path/toil-6.1.0/src/toil/resource.py [AutoAPI] Reading files... [ 7%] /build/reproducible-path/toil-6.1.0/src/toil/job.py [AutoAPI] Reading files... [ 7%] /build/reproducible-path/toil-6.1.0/src/toil/options/cwl.py [AutoAPI] Reading files... [ 8%] /build/reproducible-path/toil-6.1.0/src/toil/options/__init__.py [AutoAPI] Reading files... [ 8%] /build/reproducible-path/toil-6.1.0/src/toil/options/common.py [AutoAPI] Reading files... [ 9%] /build/reproducible-path/toil-6.1.0/src/toil/options/wdl.py [AutoAPI] Reading files... [ 9%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/torque.py [AutoAPI] Reading files... [ 10%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py [AutoAPI] Reading files... [ 10%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/gridengine.py [AutoAPI] Reading files... [ 11%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/kubernetes.py [AutoAPI] Reading files... [ 11%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/options.py [AutoAPI] Reading files... [ 12%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/lsf.py [AutoAPI] Reading files... [ 12%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py [AutoAPI] Reading files... [ 13%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/contained_executor.py [AutoAPI] Reading files... [ 13%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/__init__.py [AutoAPI] Reading files... [ 14%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/registry.py [AutoAPI] Reading files... [ 14%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/awsBatch.py [AutoAPI] Reading files... [ 15%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/local_support.py [AutoAPI] Reading files... [ 15%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractGridEngineBatchSystem.py [AutoAPI] Reading files... [ 16%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/lsfHelper.py [AutoAPI] Reading files... [ 16%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/cleanup_support.py [AutoAPI] Reading files... [ 17%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/slurm.py [AutoAPI] Reading files... [ 17%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/htcondor.py [AutoAPI] Reading files... [ 18%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/mesos/executor.py [AutoAPI] Reading files... [ 18%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/mesos/__init__.py [AutoAPI] Reading files... [ 19%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/mesos/batchSystem.py [AutoAPI] Reading files... [ 19%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/mesos/conftest.py [AutoAPI] Reading files... [ 20%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/mesos/test/__init__.py [AutoAPI] Reading files... [ 20%] /build/reproducible-path/toil-6.1.0/src/toil/test/__init__.py [AutoAPI] Reading files... [ 21%] /build/reproducible-path/toil-6.1.0/src/toil/test/options/options.py [AutoAPI] Reading files... [ 21%] /build/reproducible-path/toil-6.1.0/src/toil/test/options/__init__.py [AutoAPI] Reading files... [ 22%] /build/reproducible-path/toil-6.1.0/src/toil/test/batchSystems/batchSystemTest.py [AutoAPI] Reading files... [ 22%] /build/reproducible-path/toil-6.1.0/src/toil/test/batchSystems/test_slurm.py [AutoAPI] Reading files... [ 23%] /build/reproducible-path/toil-6.1.0/src/toil/test/batchSystems/__init__.py [AutoAPI] Reading files... [ 23%] /build/reproducible-path/toil-6.1.0/src/toil/test/batchSystems/test_lsf_helper.py [AutoAPI] Reading files... [ 24%] /build/reproducible-path/toil-6.1.0/src/toil/test/docs/__init__.py [AutoAPI] Reading files... [ 24%] /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scriptsTest.py [AutoAPI] Reading files... [ 25%] /build/reproducible-path/toil-6.1.0/src/toil/test/mesos/MesosDataStructuresTest.py [AutoAPI] Reading files... [ 25%] /build/reproducible-path/toil-6.1.0/src/toil/test/mesos/__init__.py [AutoAPI] Reading files... [ 26%] /build/reproducible-path/toil-6.1.0/src/toil/test/mesos/stress.py [AutoAPI] Reading files... [ 26%] /build/reproducible-path/toil-6.1.0/src/toil/test/mesos/helloWorld.py [AutoAPI] Reading files... [ 27%] /build/reproducible-path/toil-6.1.0/src/toil/test/wdl/wdltoil_test.py [AutoAPI] Reading files... [ 27%] /build/reproducible-path/toil-6.1.0/src/toil/test/wdl/__init__.py [AutoAPI] Reading files... [ 28%] /build/reproducible-path/toil-6.1.0/src/toil/test/cwl/__init__.py [AutoAPI] Reading files... [ 28%] /build/reproducible-path/toil-6.1.0/src/toil/test/cwl/cwlTest.py [AutoAPI] Reading files... [ 29%] /build/reproducible-path/toil-6.1.0/src/toil/test/cwl/conftest.py [AutoAPI] Reading files... [ 29%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/userDefinedJobArgTypeTest.py [AutoAPI] Reading files... [ 30%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/dockerCheckTest.py [AutoAPI] Reading files... [ 30%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/helloWorldTest.py [AutoAPI] Reading files... [ 31%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/retainTempDirTest.py [AutoAPI] Reading files... [ 31%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/importExportFileTest.py [AutoAPI] Reading files... [ 32%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/restartDAGTest.py [AutoAPI] Reading files... [ 32%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/miscTests.py [AutoAPI] Reading files... [ 33%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/autoDeploymentTest.py [AutoAPI] Reading files... [ 33%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/jobServiceTest.py [AutoAPI] Reading files... [ 34%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/promisesTest.py [AutoAPI] Reading files... [ 34%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/systemTest.py [AutoAPI] Reading files... [ 35%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/promisedRequirementTest.py [AutoAPI] Reading files... [ 35%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/realtimeLoggerTest.py [AutoAPI] Reading files... [ 36%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/jobFileStoreTest.py [AutoAPI] Reading files... [ 36%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/checkpointTest.py [AutoAPI] Reading files... [ 37%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/resumabilityTest.py [AutoAPI] Reading files... [ 37%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/__init__.py [AutoAPI] Reading files... [ 38%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/jobEncapsulationTest.py [AutoAPI] Reading files... [ 38%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/jobDescriptionTest.py [AutoAPI] Reading files... [ 39%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/toilContextManagerTest.py [AutoAPI] Reading files... [ 39%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/regularLogTest.py [AutoAPI] Reading files... [ 40%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/deferredFunctionTest.py [AutoAPI] Reading files... [ 40%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/resourceTest.py [AutoAPI] Reading files... [ 41%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/workerTest.py [AutoAPI] Reading files... [ 41%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/jobTest.py [AutoAPI] Reading files... [ 42%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/threadingTest.py [AutoAPI] Reading files... [ 42%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py [AutoAPI] Reading files... [ 43%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/busTest.py [AutoAPI] Reading files... [ 43%] /build/reproducible-path/toil-6.1.0/src/toil/test/jobStores/__init__.py [AutoAPI] Reading files... [ 44%] /build/reproducible-path/toil-6.1.0/src/toil/test/jobStores/jobStoreTest.py [AutoAPI] Reading files... [ 44%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/test_ec2.py [AutoAPI] Reading files... [ 45%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/test_conversions.py [AutoAPI] Reading files... [ 45%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/dockerTest.py [AutoAPI] Reading files... [ 46%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/test_misc.py [AutoAPI] Reading files... [ 46%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/__init__.py [AutoAPI] Reading files... [ 47%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/aws/test_s3.py [AutoAPI] Reading files... [ 47%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/aws/__init__.py [AutoAPI] Reading files... [ 48%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/aws/test_utils.py [AutoAPI] Reading files... [ 48%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/aws/test_iam.py [AutoAPI] Reading files... [ 49%] /build/reproducible-path/toil-6.1.0/src/toil/test/utils/utilsTest.py [AutoAPI] Reading files... [ 49%] /build/reproducible-path/toil-6.1.0/src/toil/test/utils/__init__.py [AutoAPI] Reading files... [ 50%] /build/reproducible-path/toil-6.1.0/src/toil/test/utils/toilDebugTest.py [AutoAPI] Reading files... [ 50%] /build/reproducible-path/toil-6.1.0/src/toil/test/utils/toilKillTest.py [AutoAPI] Reading files... [ 50%] /build/reproducible-path/toil-6.1.0/src/toil/test/sort/sort.py [AutoAPI] Reading files... [ 51%] /build/reproducible-path/toil-6.1.0/src/toil/test/sort/sortTest.py [AutoAPI] Reading files... [ 51%] /build/reproducible-path/toil-6.1.0/src/toil/test/sort/__init__.py [AutoAPI] Reading files... [ 52%] /build/reproducible-path/toil-6.1.0/src/toil/test/sort/restart_sort.py [AutoAPI] Reading files... [ 52%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/restartScript.py [AutoAPI] Reading files... [ 53%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/provisionerTest.py [AutoAPI] Reading files... [ 53%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/clusterScalerTest.py [AutoAPI] Reading files... [ 54%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/__init__.py [AutoAPI] Reading files... [ 54%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/gceProvisionerTest.py [AutoAPI] Reading files... [ 55%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/clusterTest.py [AutoAPI] Reading files... [ 55%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/aws/awsProvisionerTest.py [AutoAPI] Reading files... [ 56%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/aws/__init__.py [AutoAPI] Reading files... [ 56%] /build/reproducible-path/toil-6.1.0/src/toil/test/server/serverTest.py [AutoAPI] Reading files... [ 57%] /build/reproducible-path/toil-6.1.0/src/toil/test/server/__init__.py [AutoAPI] Reading files... [ 57%] /build/reproducible-path/toil-6.1.0/src/toil/test/cactus/__init__.py [AutoAPI] Reading files... [ 58%] /build/reproducible-path/toil-6.1.0/src/toil/test/cactus/test_cactus_integration.py [AutoAPI] Reading files... [ 58%] /build/reproducible-path/toil-6.1.0/src/toil/wdl/utils.py [AutoAPI] Reading files... [ 59%] /build/reproducible-path/toil-6.1.0/src/toil/wdl/__init__.py [AutoAPI] Reading files... [ 59%] /build/reproducible-path/toil-6.1.0/src/toil/wdl/wdltoil.py [AutoAPI] Reading files... [ 60%] /build/reproducible-path/toil-6.1.0/src/toil/cwl/utils.py [AutoAPI] Reading files... [ 60%] /build/reproducible-path/toil-6.1.0/src/toil/cwl/cwltoil.py [AutoAPI] Reading files... [ 61%] /build/reproducible-path/toil-6.1.0/src/toil/cwl/__init__.py [AutoAPI] Reading files... [ 61%] /build/reproducible-path/toil-6.1.0/src/toil/cwl/conftest.py [AutoAPI] Reading files... [ 62%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/utils.py [AutoAPI] Reading files... [ 62%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/googleJobStore.py [AutoAPI] Reading files... [ 63%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/__init__.py [AutoAPI] Reading files... [ 63%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/fileJobStore.py [AutoAPI] Reading files... [ 64%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/conftest.py [AutoAPI] Reading files... [ 64%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/abstractJobStore.py [AutoAPI] Reading files... [ 65%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/aws/utils.py [AutoAPI] Reading files... [ 65%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/aws/jobStore.py [AutoAPI] Reading files... [ 66%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/aws/__init__.py [AutoAPI] Reading files... [ 66%] /build/reproducible-path/toil-6.1.0/src/toil/lib/threading.py [AutoAPI] Reading files... [ 67%] /build/reproducible-path/toil-6.1.0/src/toil/lib/generatedEC2Lists.py [AutoAPI] Reading files... [ 67%] /build/reproducible-path/toil-6.1.0/src/toil/lib/humanize.py [AutoAPI] Reading files... [ 68%] /build/reproducible-path/toil-6.1.0/src/toil/lib/throttle.py [AutoAPI] Reading files... [ 68%] /build/reproducible-path/toil-6.1.0/src/toil/lib/compatibility.py [AutoAPI] Reading files... [ 69%] /build/reproducible-path/toil-6.1.0/src/toil/lib/objects.py [AutoAPI] Reading files... [ 69%] /build/reproducible-path/toil-6.1.0/src/toil/lib/memoize.py [AutoAPI] Reading files... [ 70%] /build/reproducible-path/toil-6.1.0/src/toil/lib/ec2.py [AutoAPI] Reading files... [ 70%] /build/reproducible-path/toil-6.1.0/src/toil/lib/accelerators.py [AutoAPI] Reading files... [ 71%] /build/reproducible-path/toil-6.1.0/src/toil/lib/exceptions.py [AutoAPI] Reading files... [ 71%] /build/reproducible-path/toil-6.1.0/src/toil/lib/expando.py [AutoAPI] Reading files... [ 72%] /build/reproducible-path/toil-6.1.0/src/toil/lib/conversions.py [AutoAPI] Reading files... [ 72%] /build/reproducible-path/toil-6.1.0/src/toil/lib/misc.py [AutoAPI] Reading files... [ 73%] /build/reproducible-path/toil-6.1.0/src/toil/lib/__init__.py [AutoAPI] Reading files... [ 73%] /build/reproducible-path/toil-6.1.0/src/toil/lib/resources.py [AutoAPI] Reading files... [ 74%] /build/reproducible-path/toil-6.1.0/src/toil/lib/retry.py [AutoAPI] Reading files... [ 74%] /build/reproducible-path/toil-6.1.0/src/toil/lib/ec2nodes.py [AutoAPI] Reading files... [ 75%] /build/reproducible-path/toil-6.1.0/src/toil/lib/io.py [AutoAPI] Reading files... [ 75%] /build/reproducible-path/toil-6.1.0/src/toil/lib/iterables.py [AutoAPI] Reading files... [ 76%] /build/reproducible-path/toil-6.1.0/src/toil/lib/bioio.py [AutoAPI] Reading files... [ 76%] /build/reproducible-path/toil-6.1.0/src/toil/lib/docker.py [AutoAPI] Reading files... [ 77%] /build/reproducible-path/toil-6.1.0/src/toil/lib/aws/utils.py [AutoAPI] Reading files... [ 77%] /build/reproducible-path/toil-6.1.0/src/toil/lib/aws/session.py [AutoAPI] Reading files... [ 78%] /build/reproducible-path/toil-6.1.0/src/toil/lib/aws/__init__.py [AutoAPI] Reading files... [ 78%] /build/reproducible-path/toil-6.1.0/src/toil/lib/aws/ami.py [AutoAPI] Reading files... [ 79%] /build/reproducible-path/toil-6.1.0/src/toil/lib/aws/iam.py [AutoAPI] Reading files... [ 79%] /build/reproducible-path/toil-6.1.0/src/toil/lib/encryption/_nacl.py [AutoAPI] Reading files... [ 80%] /build/reproducible-path/toil-6.1.0/src/toil/lib/encryption/_dummy.py [AutoAPI] Reading files... [ 80%] /build/reproducible-path/toil-6.1.0/src/toil/lib/encryption/__init__.py [AutoAPI] Reading files... [ 81%] /build/reproducible-path/toil-6.1.0/src/toil/lib/encryption/conftest.py [AutoAPI] Reading files... [ 81%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilConfig.py [AutoAPI] Reading files... [ 82%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilSshCluster.py [AutoAPI] Reading files... [ 82%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilDebugFile.py [AutoAPI] Reading files... [ 83%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilKill.py [AutoAPI] Reading files... [ 83%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilUpdateEC2Instances.py [AutoAPI] Reading files... [ 84%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilDestroyCluster.py [AutoAPI] Reading files... [ 84%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilServer.py [AutoAPI] Reading files... [ 85%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilRsyncCluster.py [AutoAPI] Reading files... [ 85%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilStats.py [AutoAPI] Reading files... [ 86%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilDebugJob.py [AutoAPI] Reading files... [ 86%] /build/reproducible-path/toil-6.1.0/src/toil/utils/__init__.py [AutoAPI] Reading files... [ 87%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilStatus.py [AutoAPI] Reading files... [ 87%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilClean.py [AutoAPI] Reading files... [ 88%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilMain.py [AutoAPI] Reading files... [ 88%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilLaunchCluster.py [AutoAPI] Reading files... [ 89%] /build/reproducible-path/toil-6.1.0/src/toil/fileStores/cachingFileStore.py [AutoAPI] Reading files... [ 89%] /build/reproducible-path/toil-6.1.0/src/toil/fileStores/__init__.py [AutoAPI] Reading files... [ 90%] /build/reproducible-path/toil-6.1.0/src/toil/fileStores/abstractFileStore.py [AutoAPI] Reading files... [ 90%] /build/reproducible-path/toil-6.1.0/src/toil/fileStores/nonCachingFileStore.py [AutoAPI] Reading files... [ 91%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/gceProvisioner.py [AutoAPI] Reading files... [ 91%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/node.py [AutoAPI] Reading files... [ 92%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/abstractProvisioner.py [AutoAPI] Reading files... [ 92%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/__init__.py [AutoAPI] Reading files... [ 93%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/clusterScaler.py [AutoAPI] Reading files... [ 93%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/aws/awsProvisioner.py [AutoAPI] Reading files... [ 94%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/aws/__init__.py [AutoAPI] Reading files... [ 94%] /build/reproducible-path/toil-6.1.0/src/toil/server/utils.py [AutoAPI] Reading files... [ 95%] /build/reproducible-path/toil-6.1.0/src/toil/server/app.py [AutoAPI] Reading files... [ 95%] /build/reproducible-path/toil-6.1.0/src/toil/server/celery_app.py [AutoAPI] Reading files... [ 96%] /build/reproducible-path/toil-6.1.0/src/toil/server/__init__.py [AutoAPI] Reading files... [ 96%] /build/reproducible-path/toil-6.1.0/src/toil/server/wsgi_app.py [AutoAPI] Reading files... [ 97%] /build/reproducible-path/toil-6.1.0/src/toil/server/wes/abstract_backend.py [AutoAPI] Reading files... [ 97%] /build/reproducible-path/toil-6.1.0/src/toil/server/wes/toil_backend.py [AutoAPI] Reading files... [ 98%] /build/reproducible-path/toil-6.1.0/src/toil/server/wes/amazon_wes_utils.py [AutoAPI] Reading files... [ 98%] /build/reproducible-path/toil-6.1.0/src/toil/server/wes/__init__.py [AutoAPI] Reading files... [ 99%] /build/reproducible-path/toil-6.1.0/src/toil/server/wes/tasks.py [AutoAPI] Reading files... [ 99%] /build/reproducible-path/toil-6.1.0/src/toil/server/cli/wes_cwl_runner.py [AutoAPI] Reading files... [100%] /build/reproducible-path/toil-6.1.0/src/toil/server/cli/__init__.py [AutoAPI] Reading files... [100%] /build/reproducible-path/toil-6.1.0/src/toil/server/api_spec/__init__.py [AutoAPI] Mapping Data... [ 0%] /build/reproducible-path/toil-6.1.0/src/toil/statsAndLogging.py [AutoAPI] Mapping Data... [ 1%] /build/reproducible-path/toil-6.1.0/src/toil/version.py [AutoAPI] Mapping Data... [ 1%] /build/reproducible-path/toil-6.1.0/src/toil/worker.py [AutoAPI] Mapping Data... [ 2%] /build/reproducible-path/toil-6.1.0/src/toil/toilState.py [AutoAPI] Mapping Data... [ 2%] /build/reproducible-path/toil-6.1.0/src/toil/realtimeLogger.py [AutoAPI] Mapping Data... [ 3%] /build/reproducible-path/toil-6.1.0/src/toil/exceptions.py [AutoAPI] Mapping Data... [ 3%] /build/reproducible-path/toil-6.1.0/src/toil/__init__.py [AutoAPI] Mapping Data... [ 4%] /build/reproducible-path/toil-6.1.0/src/toil/serviceManager.py [AutoAPI] Mapping Data... [ 4%] /build/reproducible-path/toil-6.1.0/src/toil/leader.py [AutoAPI] Mapping Data... [ 5%] /build/reproducible-path/toil-6.1.0/src/toil/bus.py [AutoAPI] Mapping Data... [ 5%] /build/reproducible-path/toil-6.1.0/src/toil/common.py [AutoAPI] Mapping Data... [ 6%] /build/reproducible-path/toil-6.1.0/src/toil/deferred.py [AutoAPI] Mapping Data... [ 6%] /build/reproducible-path/toil-6.1.0/src/toil/resource.py [AutoAPI] Mapping Data... [ 7%] /build/reproducible-path/toil-6.1.0/src/toil/job.py [AutoAPI] Mapping Data... [ 7%] /build/reproducible-path/toil-6.1.0/src/toil/options/cwl.py [AutoAPI] Mapping Data... [ 8%] /build/reproducible-path/toil-6.1.0/src/toil/options/__init__.py [AutoAPI] Mapping Data... [ 8%] /build/reproducible-path/toil-6.1.0/src/toil/options/common.py [AutoAPI] Mapping Data... [ 9%] /build/reproducible-path/toil-6.1.0/src/toil/options/wdl.py [AutoAPI] Mapping Data... [ 9%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/torque.py [AutoAPI] Mapping Data... [ 10%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py [AutoAPI] Mapping Data... [ 10%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/gridengine.py [AutoAPI] Mapping Data... [ 11%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/kubernetes.py [AutoAPI] Mapping Data... [ 11%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/options.py [AutoAPI] Mapping Data... [ 12%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/lsf.py [AutoAPI] Mapping Data... [ 12%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py [AutoAPI] Mapping Data... [ 13%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/contained_executor.py [AutoAPI] Mapping Data... [ 13%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/__init__.py [AutoAPI] Mapping Data... [ 14%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/registry.py [AutoAPI] Mapping Data... [ 14%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/awsBatch.py [AutoAPI] Mapping Data... [ 15%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/local_support.py [AutoAPI] Mapping Data... [ 15%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractGridEngineBatchSystem.py [AutoAPI] Mapping Data... [ 16%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/lsfHelper.py [AutoAPI] Mapping Data... [ 16%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/cleanup_support.py [AutoAPI] Mapping Data... [ 17%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/slurm.py [AutoAPI] Mapping Data... [ 17%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/htcondor.py [AutoAPI] Mapping Data... [ 18%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/mesos/executor.py [AutoAPI] Mapping Data... [ 18%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/mesos/__init__.py [AutoAPI] Mapping Data... [ 19%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/mesos/batchSystem.py [AutoAPI] Mapping Data... [ 19%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/mesos/conftest.py [AutoAPI] Mapping Data... [ 20%] /build/reproducible-path/toil-6.1.0/src/toil/batchSystems/mesos/test/__init__.py [AutoAPI] Mapping Data... [ 20%] /build/reproducible-path/toil-6.1.0/src/toil/test/__init__.py [AutoAPI] Mapping Data... [ 21%] /build/reproducible-path/toil-6.1.0/src/toil/test/options/options.py [AutoAPI] Mapping Data... [ 21%] /build/reproducible-path/toil-6.1.0/src/toil/test/options/__init__.py [AutoAPI] Mapping Data... [ 22%] /build/reproducible-path/toil-6.1.0/src/toil/test/batchSystems/batchSystemTest.py [AutoAPI] Mapping Data... [ 22%] /build/reproducible-path/toil-6.1.0/src/toil/test/batchSystems/test_slurm.py [AutoAPI] Mapping Data... [ 23%] /build/reproducible-path/toil-6.1.0/src/toil/test/batchSystems/__init__.py [AutoAPI] Mapping Data... [ 23%] /build/reproducible-path/toil-6.1.0/src/toil/test/batchSystems/test_lsf_helper.py [AutoAPI] Mapping Data... [ 24%] /build/reproducible-path/toil-6.1.0/src/toil/test/docs/__init__.py [AutoAPI] Mapping Data... [ 24%] /build/reproducible-path/toil-6.1.0/src/toil/test/docs/scriptsTest.py [AutoAPI] Mapping Data... [ 25%] /build/reproducible-path/toil-6.1.0/src/toil/test/mesos/MesosDataStructuresTest.py [AutoAPI] Mapping Data... [ 25%] /build/reproducible-path/toil-6.1.0/src/toil/test/mesos/__init__.py [AutoAPI] Mapping Data... [ 26%] /build/reproducible-path/toil-6.1.0/src/toil/test/mesos/stress.py [AutoAPI] Mapping Data... [ 26%] /build/reproducible-path/toil-6.1.0/src/toil/test/mesos/helloWorld.py [AutoAPI] Mapping Data... [ 27%] /build/reproducible-path/toil-6.1.0/src/toil/test/wdl/wdltoil_test.py [AutoAPI] Mapping Data... [ 27%] /build/reproducible-path/toil-6.1.0/src/toil/test/wdl/__init__.py [AutoAPI] Mapping Data... [ 28%] /build/reproducible-path/toil-6.1.0/src/toil/test/cwl/__init__.py [AutoAPI] Mapping Data... [ 28%] /build/reproducible-path/toil-6.1.0/src/toil/test/cwl/cwlTest.py [AutoAPI] Mapping Data... [ 29%] /build/reproducible-path/toil-6.1.0/src/toil/test/cwl/conftest.py [AutoAPI] Mapping Data... [ 29%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/userDefinedJobArgTypeTest.py [AutoAPI] Mapping Data... [ 30%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/dockerCheckTest.py [AutoAPI] Mapping Data... [ 30%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/helloWorldTest.py [AutoAPI] Mapping Data... [ 31%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/retainTempDirTest.py [AutoAPI] Mapping Data... [ 31%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/importExportFileTest.py [AutoAPI] Mapping Data... [ 32%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/restartDAGTest.py [AutoAPI] Mapping Data... [ 32%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/miscTests.py [AutoAPI] Mapping Data... [ 33%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/autoDeploymentTest.py [AutoAPI] Mapping Data... [ 33%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/jobServiceTest.py [AutoAPI] Mapping Data... [ 34%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/promisesTest.py [AutoAPI] Mapping Data... [ 34%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/systemTest.py [AutoAPI] Mapping Data... [ 35%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/promisedRequirementTest.py [AutoAPI] Mapping Data... [ 35%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/realtimeLoggerTest.py [AutoAPI] Mapping Data... [ 36%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/jobFileStoreTest.py [AutoAPI] Mapping Data... [ 36%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/checkpointTest.py [AutoAPI] Mapping Data... [ 37%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/resumabilityTest.py [AutoAPI] Mapping Data... [ 37%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/__init__.py [AutoAPI] Mapping Data... [ 38%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/jobEncapsulationTest.py [AutoAPI] Mapping Data... [ 38%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/jobDescriptionTest.py [AutoAPI] Mapping Data... [ 39%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/toilContextManagerTest.py [AutoAPI] Mapping Data... [ 39%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/regularLogTest.py [AutoAPI] Mapping Data... [ 40%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/deferredFunctionTest.py [AutoAPI] Mapping Data... [ 40%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/resourceTest.py [AutoAPI] Mapping Data... [ 41%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/workerTest.py [AutoAPI] Mapping Data... [ 41%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/jobTest.py [AutoAPI] Mapping Data... [ 42%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/threadingTest.py [AutoAPI] Mapping Data... [ 42%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py [AutoAPI] Mapping Data... [ 43%] /build/reproducible-path/toil-6.1.0/src/toil/test/src/busTest.py [AutoAPI] Mapping Data... [ 43%] /build/reproducible-path/toil-6.1.0/src/toil/test/jobStores/__init__.py [AutoAPI] Mapping Data... [ 44%] /build/reproducible-path/toil-6.1.0/src/toil/test/jobStores/jobStoreTest.py [AutoAPI] Mapping Data... [ 44%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/test_ec2.py [AutoAPI] Mapping Data... [ 45%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/test_conversions.py [AutoAPI] Mapping Data... [ 45%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/dockerTest.py [AutoAPI] Mapping Data... [ 46%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/test_misc.py [AutoAPI] Mapping Data... [ 46%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/__init__.py [AutoAPI] Mapping Data... [ 47%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/aws/test_s3.py [AutoAPI] Mapping Data... [ 47%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/aws/__init__.py [AutoAPI] Mapping Data... [ 48%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/aws/test_utils.py [AutoAPI] Mapping Data... [ 48%] /build/reproducible-path/toil-6.1.0/src/toil/test/lib/aws/test_iam.py [AutoAPI] Mapping Data... [ 49%] /build/reproducible-path/toil-6.1.0/src/toil/test/utils/utilsTest.py [AutoAPI] Mapping Data... [ 49%] /build/reproducible-path/toil-6.1.0/src/toil/test/utils/__init__.py [AutoAPI] Mapping Data... [ 50%] /build/reproducible-path/toil-6.1.0/src/toil/test/utils/toilDebugTest.py [AutoAPI] Mapping Data... [ 50%] /build/reproducible-path/toil-6.1.0/src/toil/test/utils/toilKillTest.py [AutoAPI] Mapping Data... [ 50%] /build/reproducible-path/toil-6.1.0/src/toil/test/sort/sort.py [AutoAPI] Mapping Data... [ 51%] /build/reproducible-path/toil-6.1.0/src/toil/test/sort/sortTest.py [AutoAPI] Mapping Data... [ 51%] /build/reproducible-path/toil-6.1.0/src/toil/test/sort/__init__.py [AutoAPI] Mapping Data... [ 52%] /build/reproducible-path/toil-6.1.0/src/toil/test/sort/restart_sort.py [AutoAPI] Mapping Data... [ 52%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/restartScript.py [AutoAPI] Mapping Data... [ 53%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/provisionerTest.py [AutoAPI] Mapping Data... [ 53%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/clusterScalerTest.py [AutoAPI] Mapping Data... [ 54%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/__init__.py [AutoAPI] Mapping Data... [ 54%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/gceProvisionerTest.py [AutoAPI] Mapping Data... [ 55%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/clusterTest.py [AutoAPI] Mapping Data... [ 55%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/aws/awsProvisionerTest.py [AutoAPI] Mapping Data... [ 56%] /build/reproducible-path/toil-6.1.0/src/toil/test/provisioners/aws/__init__.py [AutoAPI] Mapping Data... [ 56%] /build/reproducible-path/toil-6.1.0/src/toil/test/server/serverTest.py [AutoAPI] Mapping Data... [ 57%] /build/reproducible-path/toil-6.1.0/src/toil/test/server/__init__.py [AutoAPI] Mapping Data... [ 57%] /build/reproducible-path/toil-6.1.0/src/toil/test/cactus/__init__.py [AutoAPI] Mapping Data... [ 58%] /build/reproducible-path/toil-6.1.0/src/toil/test/cactus/test_cactus_integration.py [AutoAPI] Mapping Data... [ 58%] /build/reproducible-path/toil-6.1.0/src/toil/wdl/utils.py [AutoAPI] Mapping Data... [ 59%] /build/reproducible-path/toil-6.1.0/src/toil/wdl/__init__.py [AutoAPI] Mapping Data... [ 59%] /build/reproducible-path/toil-6.1.0/src/toil/wdl/wdltoil.py [AutoAPI] Mapping Data... [ 60%] /build/reproducible-path/toil-6.1.0/src/toil/cwl/utils.py [AutoAPI] Mapping Data... [ 60%] /build/reproducible-path/toil-6.1.0/src/toil/cwl/cwltoil.py [AutoAPI] Mapping Data... [ 61%] /build/reproducible-path/toil-6.1.0/src/toil/cwl/__init__.py [AutoAPI] Mapping Data... [ 61%] /build/reproducible-path/toil-6.1.0/src/toil/cwl/conftest.py [AutoAPI] Mapping Data... [ 62%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/utils.py [AutoAPI] Mapping Data... [ 62%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/googleJobStore.py [AutoAPI] Mapping Data... [ 63%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/__init__.py [AutoAPI] Mapping Data... [ 63%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/fileJobStore.py [AutoAPI] Mapping Data... [ 64%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/conftest.py [AutoAPI] Mapping Data... [ 64%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/abstractJobStore.py [AutoAPI] Mapping Data... [ 65%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/aws/utils.py [AutoAPI] Mapping Data... [ 65%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/aws/jobStore.py [AutoAPI] Mapping Data... [ 66%] /build/reproducible-path/toil-6.1.0/src/toil/jobStores/aws/__init__.py [AutoAPI] Mapping Data... [ 66%] /build/reproducible-path/toil-6.1.0/src/toil/lib/threading.py [AutoAPI] Mapping Data... [ 67%] /build/reproducible-path/toil-6.1.0/src/toil/lib/generatedEC2Lists.py [AutoAPI] Mapping Data... [ 67%] /build/reproducible-path/toil-6.1.0/src/toil/lib/humanize.py [AutoAPI] Mapping Data... [ 68%] /build/reproducible-path/toil-6.1.0/src/toil/lib/throttle.py [AutoAPI] Mapping Data... [ 68%] /build/reproducible-path/toil-6.1.0/src/toil/lib/compatibility.py [AutoAPI] Mapping Data... [ 69%] /build/reproducible-path/toil-6.1.0/src/toil/lib/objects.py [AutoAPI] Mapping Data... [ 69%] /build/reproducible-path/toil-6.1.0/src/toil/lib/memoize.py [AutoAPI] Mapping Data... [ 70%] /build/reproducible-path/toil-6.1.0/src/toil/lib/ec2.py [AutoAPI] Mapping Data... [ 70%] /build/reproducible-path/toil-6.1.0/src/toil/lib/accelerators.py [AutoAPI] Mapping Data... [ 71%] /build/reproducible-path/toil-6.1.0/src/toil/lib/exceptions.py [AutoAPI] Mapping Data... [ 71%] /build/reproducible-path/toil-6.1.0/src/toil/lib/expando.py [AutoAPI] Mapping Data... [ 72%] /build/reproducible-path/toil-6.1.0/src/toil/lib/conversions.py [AutoAPI] Mapping Data... [ 72%] /build/reproducible-path/toil-6.1.0/src/toil/lib/misc.py [AutoAPI] Mapping Data... [ 73%] /build/reproducible-path/toil-6.1.0/src/toil/lib/__init__.py [AutoAPI] Mapping Data... [ 73%] /build/reproducible-path/toil-6.1.0/src/toil/lib/resources.py [AutoAPI] Mapping Data... [ 74%] /build/reproducible-path/toil-6.1.0/src/toil/lib/retry.py [AutoAPI] Mapping Data... [ 74%] /build/reproducible-path/toil-6.1.0/src/toil/lib/ec2nodes.py [AutoAPI] Mapping Data... [ 75%] /build/reproducible-path/toil-6.1.0/src/toil/lib/io.py [AutoAPI] Mapping Data... [ 75%] /build/reproducible-path/toil-6.1.0/src/toil/lib/iterables.py [AutoAPI] Mapping Data... [ 76%] /build/reproducible-path/toil-6.1.0/src/toil/lib/bioio.py [AutoAPI] Mapping Data... [ 76%] /build/reproducible-path/toil-6.1.0/src/toil/lib/docker.py [AutoAPI] Mapping Data... [ 77%] /build/reproducible-path/toil-6.1.0/src/toil/lib/aws/utils.py [AutoAPI] Mapping Data... [ 77%] /build/reproducible-path/toil-6.1.0/src/toil/lib/aws/session.py [AutoAPI] Mapping Data... [ 78%] /build/reproducible-path/toil-6.1.0/src/toil/lib/aws/__init__.py [AutoAPI] Mapping Data... [ 78%] /build/reproducible-path/toil-6.1.0/src/toil/lib/aws/ami.py [AutoAPI] Mapping Data... [ 79%] /build/reproducible-path/toil-6.1.0/src/toil/lib/aws/iam.py [AutoAPI] Mapping Data... [ 79%] /build/reproducible-path/toil-6.1.0/src/toil/lib/encryption/_nacl.py [AutoAPI] Mapping Data... [ 80%] /build/reproducible-path/toil-6.1.0/src/toil/lib/encryption/_dummy.py [AutoAPI] Mapping Data... [ 80%] /build/reproducible-path/toil-6.1.0/src/toil/lib/encryption/__init__.py [AutoAPI] Mapping Data... [ 81%] /build/reproducible-path/toil-6.1.0/src/toil/lib/encryption/conftest.py [AutoAPI] Mapping Data... [ 81%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilConfig.py [AutoAPI] Mapping Data... [ 82%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilSshCluster.py [AutoAPI] Mapping Data... [ 82%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilDebugFile.py [AutoAPI] Mapping Data... [ 83%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilKill.py [AutoAPI] Mapping Data... [ 83%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilUpdateEC2Instances.py [AutoAPI] Mapping Data... [ 84%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilDestroyCluster.py [AutoAPI] Mapping Data... [ 84%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilServer.py [AutoAPI] Mapping Data... [ 85%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilRsyncCluster.py [AutoAPI] Mapping Data... [ 85%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilStats.py [AutoAPI] Mapping Data... [ 86%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilDebugJob.py [AutoAPI] Mapping Data... [ 86%] /build/reproducible-path/toil-6.1.0/src/toil/utils/__init__.py [AutoAPI] Mapping Data... [ 87%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilStatus.py [AutoAPI] Mapping Data... [ 87%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilClean.py [AutoAPI] Mapping Data... [ 88%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilMain.py [AutoAPI] Mapping Data... [ 88%] /build/reproducible-path/toil-6.1.0/src/toil/utils/toilLaunchCluster.py [AutoAPI] Mapping Data... [ 89%] /build/reproducible-path/toil-6.1.0/src/toil/fileStores/cachingFileStore.py [AutoAPI] Mapping Data... [ 89%] /build/reproducible-path/toil-6.1.0/src/toil/fileStores/__init__.py [AutoAPI] Mapping Data... [ 90%] /build/reproducible-path/toil-6.1.0/src/toil/fileStores/abstractFileStore.py [AutoAPI] Mapping Data... [ 90%] /build/reproducible-path/toil-6.1.0/src/toil/fileStores/nonCachingFileStore.py [AutoAPI] Mapping Data... [ 91%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/gceProvisioner.py [AutoAPI] Mapping Data... [ 91%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/node.py [AutoAPI] Mapping Data... [ 92%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/abstractProvisioner.py [AutoAPI] Mapping Data... [ 92%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/__init__.py [AutoAPI] Mapping Data... [ 93%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/clusterScaler.py [AutoAPI] Mapping Data... [ 93%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/aws/awsProvisioner.py [AutoAPI] Mapping Data... [ 94%] /build/reproducible-path/toil-6.1.0/src/toil/provisioners/aws/__init__.py [AutoAPI] Mapping Data... [ 94%] /build/reproducible-path/toil-6.1.0/src/toil/server/utils.py [AutoAPI] Mapping Data... [ 95%] /build/reproducible-path/toil-6.1.0/src/toil/server/app.py [AutoAPI] Mapping Data... [ 95%] /build/reproducible-path/toil-6.1.0/src/toil/server/celery_app.py [AutoAPI] Mapping Data... [ 96%] /build/reproducible-path/toil-6.1.0/src/toil/server/__init__.py [AutoAPI] Mapping Data... [ 96%] /build/reproducible-path/toil-6.1.0/src/toil/server/wsgi_app.py [AutoAPI] Mapping Data... [ 97%] /build/reproducible-path/toil-6.1.0/src/toil/server/wes/abstract_backend.py [AutoAPI] Mapping Data... [ 97%] /build/reproducible-path/toil-6.1.0/src/toil/server/wes/toil_backend.py [AutoAPI] Mapping Data... [ 98%] /build/reproducible-path/toil-6.1.0/src/toil/server/wes/amazon_wes_utils.py [AutoAPI] Mapping Data... [ 98%] /build/reproducible-path/toil-6.1.0/src/toil/server/wes/__init__.py [AutoAPI] Mapping Data... [ 99%] /build/reproducible-path/toil-6.1.0/src/toil/server/wes/tasks.py [AutoAPI] Mapping Data... [ 99%] /build/reproducible-path/toil-6.1.0/src/toil/server/cli/wes_cwl_runner.py [AutoAPI] Mapping Data... [100%] /build/reproducible-path/toil-6.1.0/src/toil/server/cli/__init__.py [AutoAPI] Mapping Data... [100%] /build/reproducible-path/toil-6.1.0/src/toil/server/api_spec/__init__.py [AutoAPI] Rendering Data... [ 0%] toil.statsAndLogging [AutoAPI] Rendering Data... [ 1%] toil.version [AutoAPI] Rendering Data... [ 1%] toil.worker [AutoAPI] Rendering Data... [ 2%] toil.toilState [AutoAPI] Rendering Data... [ 2%] toil.realtimeLogger [AutoAPI] Rendering Data... [ 3%] toil.exceptions [AutoAPI] Rendering Data... [ 3%] toil [AutoAPI] Rendering Data... [ 4%] toil.serviceManager [AutoAPI] Rendering Data... [ 4%] toil.leader [AutoAPI] Rendering Data... [ 5%] toil.bus [AutoAPI] Rendering Data... [ 5%] toil.common [AutoAPI] Rendering Data... [ 6%] toil.deferred [AutoAPI] Rendering Data... [ 6%] toil.resource [AutoAPI] Rendering Data... [ 7%] toil.job [AutoAPI] Rendering Data... [ 7%] toil.options.cwl [AutoAPI] Rendering Data... [ 8%] toil.options [AutoAPI] Rendering Data... [ 8%] toil.options.common [AutoAPI] Rendering Data... [ 9%] toil.options.wdl [AutoAPI] Rendering Data... [ 9%] toil.batchSystems.torque [AutoAPI] Rendering Data... [ 10%] toil.batchSystems.singleMachine [AutoAPI] Rendering Data... [ 10%] toil.batchSystems.gridengine [AutoAPI] Rendering Data... [ 11%] toil.batchSystems.kubernetes [AutoAPI] Rendering Data... [ 11%] toil.batchSystems.options [AutoAPI] Rendering Data... [ 12%] toil.batchSystems.lsf [AutoAPI] Rendering Data... [ 12%] toil.batchSystems.abstractBatchSystem [AutoAPI] Rendering Data... [ 13%] toil.batchSystems.contained_executor [AutoAPI] Rendering Data... [ 13%] toil.batchSystems [AutoAPI] Rendering Data... [ 14%] toil.batchSystems.registry [AutoAPI] Rendering Data... [ 14%] toil.batchSystems.awsBatch [AutoAPI] Rendering Data... [ 15%] toil.batchSystems.local_support [AutoAPI] Rendering Data... [ 15%] toil.batchSystems.abstractGridEngineBatchSystem [AutoAPI] Rendering Data... [ 16%] toil.batchSystems.lsfHelper [AutoAPI] Rendering Data... [ 16%] toil.batchSystems.cleanup_support [AutoAPI] Rendering Data... [ 17%] toil.batchSystems.slurm [AutoAPI] Rendering Data... [ 17%] toil.batchSystems.htcondor [AutoAPI] Rendering Data... [ 18%] toil.batchSystems.mesos.executor [AutoAPI] Rendering Data... [ 18%] toil.batchSystems.mesos [AutoAPI] Rendering Data... [ 19%] toil.batchSystems.mesos.batchSystem [AutoAPI] Rendering Data... [ 19%] toil.batchSystems.mesos.conftest [AutoAPI] Rendering Data... [ 20%] toil.batchSystems.mesos.test [AutoAPI] Rendering Data... [ 20%] toil.test [AutoAPI] Rendering Data... [ 21%] toil.test.options.options [AutoAPI] Rendering Data... [ 21%] toil.test.options [AutoAPI] Rendering Data... [ 22%] toil.test.batchSystems.batchSystemTest [AutoAPI] Rendering Data... [ 22%] toil.test.batchSystems.test_slurm [AutoAPI] Rendering Data... [ 23%] toil.test.batchSystems [AutoAPI] Rendering Data... [ 23%] toil.test.batchSystems.test_lsf_helper [AutoAPI] Rendering Data... [ 24%] toil.test.docs [AutoAPI] Rendering Data... [ 24%] toil.test.docs.scriptsTest [AutoAPI] Rendering Data... [ 25%] toil.test.mesos.MesosDataStructuresTest [AutoAPI] Rendering Data... [ 25%] toil.test.mesos [AutoAPI] Rendering Data... [ 26%] toil.test.mesos.stress [AutoAPI] Rendering Data... [ 26%] toil.test.mesos.helloWorld [AutoAPI] Rendering Data... [ 27%] toil.test.wdl.wdltoil_test [AutoAPI] Rendering Data... [ 27%] toil.test.wdl [AutoAPI] Rendering Data... [ 28%] toil.test.cwl [AutoAPI] Rendering Data... [ 28%] toil.test.cwl.cwlTest [AutoAPI] Rendering Data... [ 29%] toil.test.cwl.conftest [AutoAPI] Rendering Data... [ 29%] toil.test.src.userDefinedJobArgTypeTest [AutoAPI] Rendering Data... [ 30%] toil.test.src.dockerCheckTest [AutoAPI] Rendering Data... [ 30%] toil.test.src.helloWorldTest [AutoAPI] Rendering Data... [ 31%] toil.test.src.retainTempDirTest [AutoAPI] Rendering Data... [ 31%] toil.test.src.importExportFileTest [AutoAPI] Rendering Data... [ 32%] toil.test.src.restartDAGTest [AutoAPI] Rendering Data... [ 32%] toil.test.src.miscTests [AutoAPI] Rendering Data... [ 33%] toil.test.src.autoDeploymentTest [AutoAPI] Rendering Data... [ 33%] toil.test.src.jobServiceTest [AutoAPI] Rendering Data... [ 34%] toil.test.src.promisesTest [AutoAPI] Rendering Data... [ 34%] toil.test.src.systemTest [AutoAPI] Rendering Data... [ 35%] toil.test.src.promisedRequirementTest [AutoAPI] Rendering Data... [ 35%] toil.test.src.realtimeLoggerTest [AutoAPI] Rendering Data... [ 36%] toil.test.src.jobFileStoreTest [AutoAPI] Rendering Data... [ 36%] toil.test.src.checkpointTest [AutoAPI] Rendering Data... [ 37%] toil.test.src.resumabilityTest [AutoAPI] Rendering Data... [ 37%] toil.test.src [AutoAPI] Rendering Data... [ 38%] toil.test.src.jobEncapsulationTest [AutoAPI] Rendering Data... [ 38%] toil.test.src.jobDescriptionTest [AutoAPI] Rendering Data... [ 39%] toil.test.src.toilContextManagerTest [AutoAPI] Rendering Data... [ 39%] toil.test.src.regularLogTest [AutoAPI] Rendering Data... [ 40%] toil.test.src.deferredFunctionTest [AutoAPI] Rendering Data... [ 40%] toil.test.src.resourceTest [AutoAPI] Rendering Data... [ 41%] toil.test.src.workerTest [AutoAPI] Rendering Data... [ 41%] toil.test.src.jobTest [AutoAPI] Rendering Data... [ 42%] toil.test.src.threadingTest [AutoAPI] Rendering Data... [ 42%] toil.test.src.fileStoreTest [AutoAPI] Rendering Data... [ 43%] toil.test.src.busTest [AutoAPI] Rendering Data... [ 43%] toil.test.jobStores [AutoAPI] Rendering Data... [ 44%] toil.test.jobStores.jobStoreTest [AutoAPI] Rendering Data... [ 44%] toil.test.lib.test_ec2 [AutoAPI] Rendering Data... [ 45%] toil.test.lib.test_conversions [AutoAPI] Rendering Data... [ 45%] toil.test.lib.dockerTest [AutoAPI] Rendering Data... [ 46%] toil.test.lib.test_misc [AutoAPI] Rendering Data... [ 46%] toil.test.lib [AutoAPI] Rendering Data... [ 47%] toil.test.lib.aws.test_s3 [AutoAPI] Rendering Data... [ 47%] toil.test.lib.aws [AutoAPI] Rendering Data... [ 48%] toil.test.lib.aws.test_utils [AutoAPI] Rendering Data... [ 48%] toil.test.lib.aws.test_iam [AutoAPI] Rendering Data... [ 49%] toil.test.utils.utilsTest [AutoAPI] Rendering Data... [ 49%] toil.test.utils [AutoAPI] Rendering Data... [ 50%] toil.test.utils.toilDebugTest [AutoAPI] Rendering Data... [ 50%] toil.test.utils.toilKillTest [AutoAPI] Rendering Data... [ 50%] toil.test.sort.sort [AutoAPI] Rendering Data... [ 51%] toil.test.sort.sortTest [AutoAPI] Rendering Data... [ 51%] toil.test.sort [AutoAPI] Rendering Data... [ 52%] toil.test.sort.restart_sort [AutoAPI] Rendering Data... [ 52%] toil.test.provisioners.restartScript [AutoAPI] Rendering Data... [ 53%] toil.test.provisioners.provisionerTest [AutoAPI] Rendering Data... [ 53%] toil.test.provisioners.clusterScalerTest [AutoAPI] Rendering Data... [ 54%] toil.test.provisioners [AutoAPI] Rendering Data... [ 54%] toil.test.provisioners.gceProvisionerTest [AutoAPI] Rendering Data... [ 55%] toil.test.provisioners.clusterTest [AutoAPI] Rendering Data... [ 55%] toil.test.provisioners.aws.awsProvisionerTest [AutoAPI] Rendering Data... [ 56%] toil.test.provisioners.aws [AutoAPI] Rendering Data... [ 56%] toil.test.server.serverTest [AutoAPI] Rendering Data... [ 57%] toil.test.server [AutoAPI] Rendering Data... [ 57%] toil.test.cactus [AutoAPI] Rendering Data... [ 58%] toil.test.cactus.test_cactus_integration [AutoAPI] Rendering Data... [ 58%] toil.wdl.utils [AutoAPI] Rendering Data... [ 59%] toil.wdl [AutoAPI] Rendering Data... [ 59%] toil.wdl.wdltoil [AutoAPI] Rendering Data... [ 60%] toil.cwl.utils [AutoAPI] Rendering Data... [ 60%] toil.cwl.cwltoil [AutoAPI] Rendering Data... [ 61%] toil.cwl [AutoAPI] Rendering Data... [ 61%] toil.cwl.conftest [AutoAPI] Rendering Data... [ 62%] toil.jobStores.utils [AutoAPI] Rendering Data... [ 62%] toil.jobStores.googleJobStore [AutoAPI] Rendering Data... [ 63%] toil.jobStores [AutoAPI] Rendering Data... [ 63%] toil.jobStores.fileJobStore [AutoAPI] Rendering Data... [ 64%] toil.jobStores.conftest [AutoAPI] Rendering Data... [ 64%] toil.jobStores.abstractJobStore [AutoAPI] Rendering Data... [ 65%] toil.jobStores.aws.utils [AutoAPI] Rendering Data... [ 65%] toil.jobStores.aws.jobStore [AutoAPI] Rendering Data... [ 66%] toil.jobStores.aws [AutoAPI] Rendering Data... [ 66%] toil.lib.threading [AutoAPI] Rendering Data... [ 67%] toil.lib.generatedEC2Lists [AutoAPI] Rendering Data... [ 67%] toil.lib.humanize [AutoAPI] Rendering Data... [ 68%] toil.lib.throttle [AutoAPI] Rendering Data... [ 68%] toil.lib.compatibility [AutoAPI] Rendering Data... [ 69%] toil.lib.objects [AutoAPI] Rendering Data... [ 69%] toil.lib.memoize [AutoAPI] Rendering Data... [ 70%] toil.lib.ec2 [AutoAPI] Rendering Data... [ 70%] toil.lib.accelerators [AutoAPI] Rendering Data... [ 71%] toil.lib.exceptions [AutoAPI] Rendering Data... [ 71%] toil.lib.expando [AutoAPI] Rendering Data... [ 72%] toil.lib.conversions [AutoAPI] Rendering Data... [ 72%] toil.lib.misc [AutoAPI] Rendering Data... [ 73%] toil.lib [AutoAPI] Rendering Data... [ 73%] toil.lib.resources [AutoAPI] Rendering Data... [ 74%] toil.lib.retry [AutoAPI] Rendering Data... [ 74%] toil.lib.ec2nodes [AutoAPI] Rendering Data... [ 75%] toil.lib.io [AutoAPI] Rendering Data... [ 75%] toil.lib.iterables [AutoAPI] Rendering Data... [ 76%] toil.lib.bioio [AutoAPI] Rendering Data... [ 76%] toil.lib.docker [AutoAPI] Rendering Data... [ 77%] toil.lib.aws.utils [AutoAPI] Rendering Data... [ 77%] toil.lib.aws.session [AutoAPI] Rendering Data... [ 78%] toil.lib.aws [AutoAPI] Rendering Data... [ 78%] toil.lib.aws.ami [AutoAPI] Rendering Data... [ 79%] toil.lib.aws.iam [AutoAPI] Rendering Data... [ 79%] toil.lib.encryption._nacl [AutoAPI] Rendering Data... [ 80%] toil.lib.encryption._dummy [AutoAPI] Rendering Data... [ 80%] toil.lib.encryption [AutoAPI] Rendering Data... [ 81%] toil.lib.encryption.conftest [AutoAPI] Rendering Data... [ 81%] toil.utils.toilConfig [AutoAPI] Rendering Data... [ 82%] toil.utils.toilSshCluster [AutoAPI] Rendering Data... [ 82%] toil.utils.toilDebugFile [AutoAPI] Rendering Data... [ 83%] toil.utils.toilKill [AutoAPI] Rendering Data... [ 83%] toil.utils.toilUpdateEC2Instances [AutoAPI] Rendering Data... [ 84%] toil.utils.toilDestroyCluster [AutoAPI] Rendering Data... [ 84%] toil.utils.toilServer [AutoAPI] Rendering Data... [ 85%] toil.utils.toilRsyncCluster [AutoAPI] Rendering Data... [ 85%] toil.utils.toilStats [AutoAPI] Rendering Data... [ 86%] toil.utils.toilDebugJob [AutoAPI] Rendering Data... [ 86%] toil.utils [AutoAPI] Rendering Data... [ 87%] toil.utils.toilStatus [AutoAPI] Rendering Data... [ 87%] toil.utils.toilClean [AutoAPI] Rendering Data... [ 88%] toil.utils.toilMain [AutoAPI] Rendering Data... [ 88%] toil.utils.toilLaunchCluster [AutoAPI] Rendering Data... [ 89%] toil.fileStores.cachingFileStore [AutoAPI] Rendering Data... [ 89%] toil.fileStores [AutoAPI] Rendering Data... [ 90%] toil.fileStores.abstractFileStore [AutoAPI] Rendering Data... [ 90%] toil.fileStores.nonCachingFileStore [AutoAPI] Rendering Data... [ 91%] toil.provisioners.gceProvisioner [AutoAPI] Rendering Data... [ 91%] toil.provisioners.node [AutoAPI] Rendering Data... [ 92%] toil.provisioners.abstractProvisioner [AutoAPI] Rendering Data... [ 92%] toil.provisioners [AutoAPI] Rendering Data... [ 93%] toil.provisioners.clusterScaler [AutoAPI] Rendering Data... [ 93%] toil.provisioners.aws.awsProvisioner [AutoAPI] Rendering Data... [ 94%] toil.provisioners.aws [AutoAPI] Rendering Data... [ 94%] toil.server.utils [AutoAPI] Rendering Data... [ 95%] toil.server.app [AutoAPI] Rendering Data... [ 95%] toil.server.celery_app [AutoAPI] Rendering Data... [ 96%] toil.server [AutoAPI] Rendering Data... [ 96%] toil.server.wsgi_app [AutoAPI] Rendering Data... [ 97%] toil.server.wes.abstract_backend [AutoAPI] Rendering Data... [ 97%] toil.server.wes.toil_backend [AutoAPI] Rendering Data... [ 98%] toil.server.wes.amazon_wes_utils [AutoAPI] Rendering Data... [ 98%] toil.server.wes [AutoAPI] Rendering Data... [ 99%] toil.server.wes.tasks [AutoAPI] Rendering Data... [ 99%] toil.server.cli.wes_cwl_runner [AutoAPI] Rendering Data... [100%] toil.server.cli [AutoAPI] Rendering Data... [100%] toil.server.api_spec loading intersphinx inventory from /usr/share/doc/python3.11/html/objects.inv... loading intersphinx inventory from https://cwltool.readthedocs.io/en/latest/objects.inv... WARNING: failed to reach any of the inventories with the following issues: intersphinx inventory 'https://cwltool.readthedocs.io/en/latest/objects.inv' not fetchable due to : HTTPSConnectionPool(host='cwltool.readthedocs.io', port=443): Max retries exceeded with url: /en/latest/objects.inv (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) building [mo]: targets for 0 po files that are out of date writing output... building [man]: all manpages updating environment: [new config] 239 added, 0 changed, 0 removed reading sources... [ 0%] appendices/architecture reading sources... [ 1%] appendices/aws_min_permissions reading sources... [ 1%] appendices/deploy reading sources... [ 2%] appendices/environment_vars reading sources... [ 2%] autoapi/index reading sources... [ 3%] autoapi/toil/batchSystems/abstractBatchSystem/index reading sources... [ 3%] autoapi/toil/batchSystems/abstractGridEngineBatchSystem/index reading sources... [ 3%] autoapi/toil/batchSystems/awsBatch/index reading sources... [ 4%] autoapi/toil/batchSystems/cleanup_support/index reading sources... [ 4%] autoapi/toil/batchSystems/contained_executor/index reading sources... [ 5%] autoapi/toil/batchSystems/gridengine/index reading sources... [ 5%] autoapi/toil/batchSystems/htcondor/index reading sources... [ 5%] autoapi/toil/batchSystems/index reading sources... [ 6%] autoapi/toil/batchSystems/kubernetes/index reading sources... [ 6%] autoapi/toil/batchSystems/local_support/index reading sources... [ 7%] autoapi/toil/batchSystems/lsf/index reading sources... [ 7%] autoapi/toil/batchSystems/lsfHelper/index reading sources... [ 8%] autoapi/toil/batchSystems/mesos/batchSystem/index reading sources... [ 8%] autoapi/toil/batchSystems/mesos/conftest/index reading sources... [ 8%] autoapi/toil/batchSystems/mesos/executor/index reading sources... [ 9%] autoapi/toil/batchSystems/mesos/index reading sources... [ 9%] autoapi/toil/batchSystems/mesos/test/index reading sources... [ 10%] autoapi/toil/batchSystems/options/index reading sources... [ 10%] autoapi/toil/batchSystems/registry/index reading sources... [ 10%] autoapi/toil/batchSystems/singleMachine/index reading sources... [ 11%] autoapi/toil/batchSystems/slurm/index reading sources... [ 11%] autoapi/toil/batchSystems/torque/index reading sources... [ 12%] autoapi/toil/bus/index reading sources... [ 12%] autoapi/toil/common/index reading sources... [ 13%] autoapi/toil/cwl/conftest/index /usr/lib/python3/dist-packages/cwltool/__init__.py reading sources... [ 13%] autoapi/toil/cwl/cwltoil/index WARNING: You are using cwltool version 3.1.20240404144621, which is not the version Toil is tested against. To install the correct cwltool for Toil, do: pip install cwltool==3.1.20240112164112 reading sources... [ 13%] autoapi/toil/cwl/index reading sources... [ 14%] autoapi/toil/cwl/utils/index reading sources... [ 14%] autoapi/toil/deferred/index reading sources... [ 15%] autoapi/toil/exceptions/index reading sources... [ 15%] autoapi/toil/fileStores/abstractFileStore/index reading sources... [ 15%] autoapi/toil/fileStores/cachingFileStore/index reading sources... [ 16%] autoapi/toil/fileStores/index reading sources... [ 16%] autoapi/toil/fileStores/nonCachingFileStore/index reading sources... [ 17%] autoapi/toil/index reading sources... [ 17%] autoapi/toil/job/index reading sources... [ 18%] autoapi/toil/jobStores/abstractJobStore/index reading sources... [ 18%] autoapi/toil/jobStores/aws/index reading sources... [ 18%] autoapi/toil/jobStores/aws/jobStore/index reading sources... [ 19%] autoapi/toil/jobStores/aws/utils/index reading sources... [ 19%] autoapi/toil/jobStores/conftest/index reading sources... [ 20%] autoapi/toil/jobStores/fileJobStore/index reading sources... [ 20%] autoapi/toil/jobStores/googleJobStore/index reading sources... [ 21%] autoapi/toil/jobStores/index reading sources... [ 21%] autoapi/toil/jobStores/utils/index reading sources... [ 21%] autoapi/toil/leader/index reading sources... [ 22%] autoapi/toil/lib/accelerators/index reading sources... [ 22%] autoapi/toil/lib/aws/ami/index reading sources... [ 23%] autoapi/toil/lib/aws/iam/index reading sources... [ 23%] autoapi/toil/lib/aws/index reading sources... [ 23%] autoapi/toil/lib/aws/session/index reading sources... [ 24%] autoapi/toil/lib/aws/utils/index reading sources... [ 24%] autoapi/toil/lib/bioio/index reading sources... [ 25%] autoapi/toil/lib/compatibility/index reading sources... [ 25%] autoapi/toil/lib/conversions/index reading sources... [ 26%] autoapi/toil/lib/docker/index reading sources... [ 26%] autoapi/toil/lib/ec2/index reading sources... [ 26%] autoapi/toil/lib/ec2nodes/index reading sources... [ 27%] autoapi/toil/lib/encryption/_dummy/index reading sources... [ 27%] autoapi/toil/lib/encryption/_nacl/index reading sources... [ 28%] autoapi/toil/lib/encryption/conftest/index reading sources... [ 28%] autoapi/toil/lib/encryption/index reading sources... [ 28%] autoapi/toil/lib/exceptions/index reading sources... [ 29%] autoapi/toil/lib/expando/index reading sources... [ 29%] autoapi/toil/lib/generatedEC2Lists/index reading sources... [ 30%] autoapi/toil/lib/humanize/index reading sources... [ 30%] autoapi/toil/lib/index reading sources... [ 31%] autoapi/toil/lib/io/index reading sources... [ 31%] autoapi/toil/lib/iterables/index reading sources... [ 31%] autoapi/toil/lib/memoize/index reading sources... [ 32%] autoapi/toil/lib/misc/index reading sources... [ 32%] autoapi/toil/lib/objects/index reading sources... [ 33%] autoapi/toil/lib/resources/index reading sources... [ 33%] autoapi/toil/lib/retry/index reading sources... [ 33%] autoapi/toil/lib/threading/index reading sources... [ 34%] autoapi/toil/lib/throttle/index reading sources... [ 34%] autoapi/toil/options/common/index reading sources... [ 35%] autoapi/toil/options/cwl/index reading sources... [ 35%] autoapi/toil/options/index reading sources... [ 36%] autoapi/toil/options/wdl/index reading sources... [ 36%] autoapi/toil/provisioners/abstractProvisioner/index reading sources... [ 36%] autoapi/toil/provisioners/aws/awsProvisioner/index reading sources... [ 37%] autoapi/toil/provisioners/aws/index reading sources... [ 37%] autoapi/toil/provisioners/clusterScaler/index reading sources... [ 38%] autoapi/toil/provisioners/gceProvisioner/index reading sources... [ 38%] autoapi/toil/provisioners/index reading sources... [ 38%] autoapi/toil/provisioners/node/index reading sources... [ 39%] autoapi/toil/realtimeLogger/index reading sources... [ 39%] autoapi/toil/resource/index reading sources... [ 40%] autoapi/toil/server/api_spec/index reading sources... [ 40%] autoapi/toil/server/app/index reading sources... [ 41%] autoapi/toil/server/celery_app/index reading sources... [ 41%] autoapi/toil/server/cli/index reading sources... [ 41%] autoapi/toil/server/cli/wes_cwl_runner/index reading sources... [ 42%] autoapi/toil/server/index reading sources... [ 42%] autoapi/toil/server/utils/index reading sources... [ 43%] autoapi/toil/server/wes/abstract_backend/index reading sources... [ 43%] autoapi/toil/server/wes/amazon_wes_utils/index reading sources... [ 44%] autoapi/toil/server/wes/index reading sources... [ 44%] autoapi/toil/server/wes/tasks/index reading sources... [ 44%] autoapi/toil/server/wes/toil_backend/index reading sources... [ 45%] autoapi/toil/server/wsgi_app/index reading sources... [ 45%] autoapi/toil/serviceManager/index reading sources... [ 46%] autoapi/toil/statsAndLogging/index reading sources... [ 46%] autoapi/toil/test/batchSystems/batchSystemTest/index reading sources... [ 46%] autoapi/toil/test/batchSystems/index reading sources... [ 47%] autoapi/toil/test/batchSystems/test_lsf_helper/index reading sources... [ 47%] autoapi/toil/test/batchSystems/test_slurm/index reading sources... [ 48%] autoapi/toil/test/cactus/index reading sources... [ 48%] autoapi/toil/test/cactus/test_cactus_integration/index reading sources... [ 49%] autoapi/toil/test/cwl/conftest/index reading sources... [ 49%] autoapi/toil/test/cwl/cwlTest/index docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. See 'docker run --help'. reading sources... [ 49%] autoapi/toil/test/cwl/index reading sources... [ 50%] autoapi/toil/test/docs/index reading sources... [ 50%] autoapi/toil/test/docs/scriptsTest/index reading sources... [ 51%] autoapi/toil/test/index reading sources... [ 51%] autoapi/toil/test/jobStores/index reading sources... [ 51%] autoapi/toil/test/jobStores/jobStoreTest/index reading sources... [ 52%] autoapi/toil/test/lib/aws/index reading sources... [ 52%] autoapi/toil/test/lib/aws/test_iam/index reading sources... [ 53%] autoapi/toil/test/lib/aws/test_s3/index reading sources... [ 53%] autoapi/toil/test/lib/aws/test_utils/index reading sources... [ 54%] autoapi/toil/test/lib/dockerTest/index reading sources... [ 54%] autoapi/toil/test/lib/index reading sources... [ 54%] autoapi/toil/test/lib/test_conversions/index reading sources... [ 55%] autoapi/toil/test/lib/test_ec2/index reading sources... [ 55%] autoapi/toil/test/lib/test_misc/index reading sources... [ 56%] autoapi/toil/test/mesos/MesosDataStructuresTest/index reading sources... [ 56%] autoapi/toil/test/mesos/helloWorld/index reading sources... [ 56%] autoapi/toil/test/mesos/index reading sources... [ 57%] autoapi/toil/test/mesos/stress/index reading sources... [ 57%] autoapi/toil/test/options/index reading sources... [ 58%] autoapi/toil/test/options/options/index reading sources... [ 58%] autoapi/toil/test/provisioners/aws/awsProvisionerTest/index reading sources... [ 59%] autoapi/toil/test/provisioners/aws/index reading sources... [ 59%] autoapi/toil/test/provisioners/clusterScalerTest/index reading sources... [ 59%] autoapi/toil/test/provisioners/clusterTest/index reading sources... [ 60%] autoapi/toil/test/provisioners/gceProvisionerTest/index INFO:toil:Using default docker registry of quay.io/ucsc_cgl as TOIL_DOCKER_REGISTRY is not set. INFO:toil:Using default docker name of toil as TOIL_DOCKER_NAME is not set. INFO:toil:Using default docker appliance of quay.io/ucsc_cgl/toil:6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f-py3.11 as TOIL_APPLIANCE_SELF is not set. reading sources... [ 60%] autoapi/toil/test/provisioners/index reading sources... [ 61%] autoapi/toil/test/provisioners/provisionerTest/index reading sources... [ 61%] autoapi/toil/test/provisioners/restartScript/index reading sources... [ 62%] autoapi/toil/test/server/index reading sources... [ 62%] autoapi/toil/test/server/serverTest/index reading sources... [ 62%] autoapi/toil/test/sort/index reading sources... [ 63%] autoapi/toil/test/sort/restart_sort/index reading sources... [ 63%] autoapi/toil/test/sort/sort/index reading sources... [ 64%] autoapi/toil/test/sort/sortTest/index reading sources... [ 64%] autoapi/toil/test/src/autoDeploymentTest/index INFO:toil:Using default docker registry of quay.io/ucsc_cgl as TOIL_DOCKER_REGISTRY is not set. INFO:toil:Using default docker name of toil as TOIL_DOCKER_NAME is not set. INFO:toil:Using default docker appliance of quay.io/ucsc_cgl/toil:6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f-py3.11 as TOIL_APPLIANCE_SELF is not set. reading sources... [ 64%] autoapi/toil/test/src/busTest/index reading sources... [ 65%] autoapi/toil/test/src/checkpointTest/index reading sources... [ 65%] autoapi/toil/test/src/deferredFunctionTest/index reading sources... [ 66%] autoapi/toil/test/src/dockerCheckTest/index reading sources... [ 66%] autoapi/toil/test/src/fileStoreTest/index reading sources... [ 67%] autoapi/toil/test/src/helloWorldTest/index reading sources... [ 67%] autoapi/toil/test/src/importExportFileTest/index reading sources... [ 67%] autoapi/toil/test/src/index reading sources... [ 68%] autoapi/toil/test/src/jobDescriptionTest/index reading sources... [ 68%] autoapi/toil/test/src/jobEncapsulationTest/index reading sources... [ 69%] autoapi/toil/test/src/jobFileStoreTest/index reading sources... [ 69%] autoapi/toil/test/src/jobServiceTest/index reading sources... [ 69%] autoapi/toil/test/src/jobTest/index reading sources... [ 70%] autoapi/toil/test/src/miscTests/index reading sources... [ 70%] autoapi/toil/test/src/promisedRequirementTest/index reading sources... [ 71%] autoapi/toil/test/src/promisesTest/index reading sources... [ 71%] autoapi/toil/test/src/realtimeLoggerTest/index reading sources... [ 72%] autoapi/toil/test/src/regularLogTest/index reading sources... [ 72%] autoapi/toil/test/src/resourceTest/index reading sources... [ 72%] autoapi/toil/test/src/restartDAGTest/index reading sources... [ 73%] autoapi/toil/test/src/resumabilityTest/index reading sources... [ 73%] autoapi/toil/test/src/retainTempDirTest/index reading sources... [ 74%] autoapi/toil/test/src/systemTest/index reading sources... [ 74%] autoapi/toil/test/src/threadingTest/index reading sources... [ 74%] autoapi/toil/test/src/toilContextManagerTest/index reading sources... [ 75%] autoapi/toil/test/src/userDefinedJobArgTypeTest/index reading sources... [ 75%] autoapi/toil/test/src/workerTest/index reading sources... [ 76%] autoapi/toil/test/utils/index reading sources... [ 76%] autoapi/toil/test/utils/toilDebugTest/index reading sources... [ 77%] autoapi/toil/test/utils/toilKillTest/index reading sources... [ 77%] autoapi/toil/test/utils/utilsTest/index reading sources... [ 77%] autoapi/toil/test/wdl/index reading sources... [ 78%] autoapi/toil/test/wdl/wdltoil_test/index reading sources... [ 78%] autoapi/toil/toilState/index reading sources... [ 79%] autoapi/toil/utils/index reading sources... [ 79%] autoapi/toil/utils/toilClean/index reading sources... [ 79%] autoapi/toil/utils/toilConfig/index reading sources... [ 80%] autoapi/toil/utils/toilDebugFile/index reading sources... [ 80%] autoapi/toil/utils/toilDebugJob/index reading sources... [ 81%] autoapi/toil/utils/toilDestroyCluster/index reading sources... [ 81%] autoapi/toil/utils/toilKill/index reading sources... [ 82%] autoapi/toil/utils/toilLaunchCluster/index reading sources... [ 82%] autoapi/toil/utils/toilMain/index reading sources... [ 82%] autoapi/toil/utils/toilRsyncCluster/index reading sources... [ 83%] autoapi/toil/utils/toilServer/index reading sources... [ 83%] autoapi/toil/utils/toilSshCluster/index reading sources... [ 84%] autoapi/toil/utils/toilStats/index reading sources... [ 84%] autoapi/toil/utils/toilStatus/index reading sources... [ 85%] autoapi/toil/utils/toilUpdateEC2Instances/index reading sources... [ 85%] autoapi/toil/version/index reading sources... [ 85%] autoapi/toil/wdl/index reading sources... [ 86%] autoapi/toil/wdl/utils/index reading sources... [ 86%] autoapi/toil/wdl/wdltoil/index reading sources... [ 87%] autoapi/toil/worker/index reading sources... [ 87%] contributing/checklists reading sources... [ 87%] contributing/contributing reading sources... [ 88%] cwl/introduction reading sources... [ 88%] cwl/running reading sources... [ 89%] gettingStarted/install reading sources... [ 89%] gettingStarted/quickStart reading sources... [ 90%] index [AutoAPI] Adding AutoAPI TOCTree [autoapi/index] to index.rst reading sources... [ 90%] python/developing reading sources... [ 90%] python/toilAPI reading sources... [ 91%] python/toilAPIBatchsystem reading sources... [ 91%] python/toilAPIExceptions reading sources... [ 92%] python/toilAPIFilestore reading sources... [ 92%] python/toilAPIJobFunctions reading sources... [ 92%] python/toilAPIJobstore reading sources... [ 93%] python/toilAPIMethods reading sources... [ 93%] python/toilAPIRunner reading sources... [ 94%] python/toilAPIService reading sources... [ 94%] running/cliOptions reading sources... [ 95%] running/cloud/amazon reading sources... [ 95%] running/cloud/cloud reading sources... [ 95%] running/cloud/clusterUtils reading sources... [ 96%] running/cloud/gce reading sources... [ 96%] running/cloud/kubernetes reading sources... [ 97%] running/debugging reading sources... [ 97%] running/hpcEnvironments reading sources... [ 97%] running/introduction reading sources... [ 98%] running/server/wes reading sources... [ 98%] running/utils reading sources... [ 99%] wdl/conformance reading sources... [ 99%] wdl/developing reading sources... [100%] wdl/introduction reading sources... [100%] wdl/running looking for now-outdated files... none found pickling environment... done checking consistency... /build/reproducible-path/toil-6.1.0/docs/gettingStarted/install.rst: WARNING: document isn't included in any toctree done writing... toil.1 { gettingStarted/quickStart cwl/introduction cwl/running wdl/introduction wdl/running wdl/developing wdl/conformance running/introduction running/cliOptions running/utils running/debugging running/cloud/cloud running/cloud/clusterUtils running/cloud/kubernetes running/cloud/amazon running/cloud/gce running/hpcEnvironments running/server/wes python/developing python/toilAPI python/toilAPIJobstore python/toilAPIJobFunctions python/toilAPIMethods python/toilAPIRunner python/toilAPIFilestore python/toilAPIBatchsystem python/toilAPIService python/toilAPIExceptions contributing/contributing contributing/checklists appendices/architecture appendices/aws_min_permissions appendices/deploy appendices/environment_vars autoapi/index autoapi/toil/index autoapi/toil/batchSystems/index autoapi/toil/batchSystems/mesos/index autoapi/toil/batchSystems/mesos/test/index autoapi/toil/batchSystems/mesos/batchSystem/index autoapi/toil/batchSystems/mesos/conftest/index autoapi/toil/batchSystems/mesos/executor/index autoapi/toil/batchSystems/abstractBatchSystem/index autoapi/toil/batchSystems/abstractGridEngineBatchSystem/index autoapi/toil/batchSystems/awsBatch/index autoapi/toil/batchSystems/cleanup_support/index autoapi/toil/batchSystems/contained_executor/index autoapi/toil/batchSystems/gridengine/index autoapi/toil/batchSystems/htcondor/index autoapi/toil/batchSystems/kubernetes/index autoapi/toil/batchSystems/local_support/index autoapi/toil/batchSystems/lsf/index autoapi/toil/batchSystems/lsfHelper/index autoapi/toil/batchSystems/options/index autoapi/toil/batchSystems/registry/index autoapi/toil/batchSystems/singleMachine/index autoapi/toil/batchSystems/slurm/index autoapi/toil/batchSystems/torque/index autoapi/toil/cwl/index autoapi/toil/cwl/conftest/index autoapi/toil/cwl/cwltoil/index autoapi/toil/cwl/utils/index autoapi/toil/fileStores/index autoapi/toil/fileStores/abstractFileStore/index autoapi/toil/fileStores/cachingFileStore/index autoapi/toil/fileStores/nonCachingFileStore/index autoapi/toil/jobStores/index autoapi/toil/jobStores/aws/index autoapi/toil/jobStores/aws/jobStore/index autoapi/toil/jobStores/aws/utils/index autoapi/toil/jobStores/abstractJobStore/index autoapi/toil/jobStores/conftest/index autoapi/toil/jobStores/fileJobStore/index autoapi/toil/jobStores/googleJobStore/index autoapi/toil/jobStores/utils/index autoapi/toil/lib/index autoapi/toil/lib/aws/index autoapi/toil/lib/aws/ami/index autoapi/toil/lib/aws/iam/index autoapi/toil/lib/aws/session/index autoapi/toil/lib/aws/utils/index autoapi/toil/lib/encryption/index autoapi/toil/lib/encryption/conftest/index autoapi/toil/lib/accelerators/index autoapi/toil/lib/bioio/index autoapi/toil/lib/compatibility/index autoapi/toil/lib/conversions/index autoapi/toil/lib/docker/index autoapi/toil/lib/ec2/index autoapi/toil/lib/ec2nodes/index autoapi/toil/lib/exceptions/index autoapi/toil/lib/expando/index autoapi/toil/lib/generatedEC2Lists/index autoapi/toil/lib/humanize/index autoapi/toil/lib/io/index autoapi/toil/lib/iterables/index autoapi/toil/lib/memoize/index autoapi/toil/lib/misc/index autoapi/toil/lib/objects/index autoapi/toil/lib/resources/index autoapi/toil/lib/retry/index autoapi/toil/lib/threading/index autoapi/toil/lib/throttle/index autoapi/toil/options/index autoapi/toil/options/common/index autoapi/toil/options/cwl/index autoapi/toil/options/wdl/index autoapi/toil/provisioners/index autoapi/toil/provisioners/aws/index autoapi/toil/provisioners/aws/awsProvisioner/index autoapi/toil/provisioners/abstractProvisioner/index autoapi/toil/provisioners/clusterScaler/index autoapi/toil/provisioners/gceProvisioner/index autoapi/toil/provisioners/node/index autoapi/toil/server/index autoapi/toil/server/api_spec/index autoapi/toil/server/cli/index autoapi/toil/server/cli/wes_cwl_runner/index autoapi/toil/server/wes/index autoapi/toil/server/wes/abstract_backend/index autoapi/toil/server/wes/amazon_wes_utils/index autoapi/toil/server/wes/tasks/index autoapi/toil/server/wes/toil_backend/index autoapi/toil/server/app/index autoapi/toil/server/celery_app/index autoapi/toil/server/utils/index autoapi/toil/server/wsgi_app/index autoapi/toil/test/index autoapi/toil/test/batchSystems/index autoapi/toil/test/batchSystems/batchSystemTest/index autoapi/toil/test/batchSystems/test_lsf_helper/index autoapi/toil/test/batchSystems/test_slurm/index autoapi/toil/test/cactus/index autoapi/toil/test/cactus/test_cactus_integration/index autoapi/toil/test/cwl/index autoapi/toil/test/cwl/conftest/index autoapi/toil/test/cwl/cwlTest/index autoapi/toil/test/docs/index autoapi/toil/test/docs/scriptsTest/index autoapi/toil/test/jobStores/index autoapi/toil/test/jobStores/jobStoreTest/index autoapi/toil/test/lib/index autoapi/toil/test/lib/aws/index autoapi/toil/test/lib/aws/test_iam/index autoapi/toil/test/lib/aws/test_s3/index autoapi/toil/test/lib/aws/test_utils/index autoapi/toil/test/lib/dockerTest/index autoapi/toil/test/lib/test_conversions/index autoapi/toil/test/lib/test_ec2/index autoapi/toil/test/lib/test_misc/index autoapi/toil/test/mesos/index autoapi/toil/test/mesos/MesosDataStructuresTest/index autoapi/toil/test/mesos/helloWorld/index autoapi/toil/test/mesos/stress/index autoapi/toil/test/options/index autoapi/toil/test/options/options/index autoapi/toil/test/provisioners/index autoapi/toil/test/provisioners/aws/index autoapi/toil/test/provisioners/aws/awsProvisionerTest/index autoapi/toil/test/provisioners/clusterScalerTest/index autoapi/toil/test/provisioners/clusterTest/index autoapi/toil/test/provisioners/gceProvisionerTest/index autoapi/toil/test/provisioners/provisionerTest/index autoapi/toil/test/provisioners/restartScript/index autoapi/toil/test/server/index autoapi/toil/test/server/serverTest/index autoapi/toil/test/sort/index autoapi/toil/test/sort/restart_sort/index autoapi/toil/test/sort/sort/index autoapi/toil/test/sort/sortTest/index autoapi/toil/test/src/index autoapi/toil/test/src/autoDeploymentTest/index autoapi/toil/test/src/busTest/index autoapi/toil/test/src/checkpointTest/index autoapi/toil/test/src/deferredFunctionTest/index autoapi/toil/test/src/dockerCheckTest/index autoapi/toil/test/src/fileStoreTest/index autoapi/toil/test/src/helloWorldTest/index autoapi/toil/test/src/importExportFileTest/index autoapi/toil/test/src/jobDescriptionTest/index autoapi/toil/test/src/jobEncapsulationTest/index autoapi/toil/test/src/jobFileStoreTest/index autoapi/toil/test/src/jobServiceTest/index autoapi/toil/test/src/jobTest/index autoapi/toil/test/src/miscTests/index autoapi/toil/test/src/promisedRequirementTest/index autoapi/toil/test/src/promisesTest/index autoapi/toil/test/src/realtimeLoggerTest/index autoapi/toil/test/src/regularLogTest/index autoapi/toil/test/src/resourceTest/index autoapi/toil/test/src/restartDAGTest/index autoapi/toil/test/src/resumabilityTest/index autoapi/toil/test/src/retainTempDirTest/index autoapi/toil/test/src/systemTest/index autoapi/toil/test/src/threadingTest/index autoapi/toil/test/src/toilContextManagerTest/index autoapi/toil/test/src/userDefinedJobArgTypeTest/index autoapi/toil/test/src/workerTest/index autoapi/toil/test/utils/index autoapi/toil/test/utils/toilDebugTest/index autoapi/toil/test/utils/toilKillTest/index autoapi/toil/test/utils/utilsTest/index autoapi/toil/test/wdl/index autoapi/toil/test/wdl/wdltoil_test/index autoapi/toil/utils/index autoapi/toil/utils/toilClean/index autoapi/toil/utils/toilConfig/index autoapi/toil/utils/toilDebugFile/index autoapi/toil/utils/toilDebugJob/index autoapi/toil/utils/toilDestroyCluster/index autoapi/toil/utils/toilKill/index autoapi/toil/utils/toilLaunchCluster/index autoapi/toil/utils/toilMain/index autoapi/toil/utils/toilRsyncCluster/index autoapi/toil/utils/toilServer/index autoapi/toil/utils/toilSshCluster/index autoapi/toil/utils/toilStats/index autoapi/toil/utils/toilStatus/index autoapi/toil/utils/toilUpdateEC2Instances/index autoapi/toil/wdl/index autoapi/toil/wdl/utils/index autoapi/toil/wdl/wdltoil/index autoapi/toil/bus/index autoapi/toil/common/index autoapi/toil/deferred/index autoapi/toil/exceptions/index autoapi/toil/job/index autoapi/toil/leader/index autoapi/toil/realtimeLogger/index autoapi/toil/resource/index autoapi/toil/serviceManager/index autoapi/toil/statsAndLogging/index autoapi/toil/toilState/index autoapi/toil/version/index autoapi/toil/worker/index } /build/reproducible-path/toil-6.1.0/docs/gettingStarted/quickStart.rst:506: WARNING: "table row spanning" not supported done build succeeded, 3 warnings. The manual pages are in _build/man. make[2]: Leaving directory '/build/reproducible-path/toil-6.1.0/docs' make[1]: Leaving directory '/build/reproducible-path/toil-6.1.0' debian/rules override_dh_auto_test make[1]: Entering directory '/build/reproducible-path/toil-6.1.0' PYBUILD_SYSTEM=custom \ PYBUILD_TEST_ARGS='HOME={home_dir} {interpreter} setup.py develop --user && PYTHONPATH={dir}/src:$PYTHONPATH PATH={home_dir}/.local/bin/:$PATH TOIL_TEST_QUICK=True TOIL_SKIP_DOCKER=True {interpreter} -m pytest -n auto --dist loadscope -vv -W ignore --ignore src/toil/test/lib/aws/test_s3.py --ignore src/toil/test/provisioners/aws/awsProvisionerTest.py --ignore src/toil/test/wdl/wdltoil_test.py --ignore src/toil/test/cwl/cwlTest.py --ignore src/toil/test/src/promisedRequirementTest.py --ignore src/toil/test/lib/test_ec2.py --ignore src/toil/test/batchSystems/batchSystemTest.py --ignore src/toil/test/lib/aws/test_iam.py -k "not (test_bioconda or test_run_conformance or testImportFtpFile or ToilWdlIntegrationTest or SortTest or testCwlexample or testVirtualEnv or ToilDocumentationTest or test_cwl_toil_kill or testImportReadFileCompatibility or CleanWorkDirTest or DeferredFunctionTest or WdlToilTest or test_cactus_integration)" {dir}/src/toil/test' \ dh_auto_test I: pybuild base:311: HOME=/build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil python3.12 setup.py develop --user && PYTHONPATH=/build/reproducible-path/toil-6.1.0/src:$PYTHONPATH PATH=/build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/.local/bin/:$PATH TOIL_TEST_QUICK=True TOIL_SKIP_DOCKER=True python3.12 -m pytest -n auto --dist loadscope -vv -W ignore --ignore src/toil/test/lib/aws/test_s3.py --ignore src/toil/test/provisioners/aws/awsProvisionerTest.py --ignore src/toil/test/wdl/wdltoil_test.py --ignore src/toil/test/cwl/cwlTest.py --ignore src/toil/test/src/promisedRequirementTest.py --ignore src/toil/test/lib/test_ec2.py --ignore src/toil/test/batchSystems/batchSystemTest.py --ignore src/toil/test/lib/aws/test_iam.py -k "not (test_bioconda or test_run_conformance or testImportFtpFile or ToilWdlIntegrationTest or SortTest or testCwlexample or testVirtualEnv or ToilDocumentationTest or test_cwl_toil_kill or testImportReadFileCompatibility or CleanWorkDirTest or DeferredFunctionTest or WdlToilTest or test_cactus_integration)" /build/reproducible-path/toil-6.1.0/src/toil/test running develop /usr/lib/python3/dist-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated. !! ******************************************************************************** Please avoid running ``setup.py`` and ``easy_install``. Instead, use pypa/build, pypa/installer or other standards-based tools. See https://github.com/pypa/setuptools/issues/917 for details. ******************************************************************************** !! easy_install.initialize_options(self) /usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated. !! ******************************************************************************** Please avoid running ``setup.py`` directly. Instead, use pypa/build, pypa/installer or other standards-based tools. See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details. ******************************************************************************** !! self.initialize_options() running egg_info creating src/toil.egg-info writing src/toil.egg-info/PKG-INFO writing dependency_links to src/toil.egg-info/dependency_links.txt writing entry points to src/toil.egg-info/entry_points.txt writing requirements to src/toil.egg-info/requires.txt writing top-level names to src/toil.egg-info/top_level.txt writing manifest file 'src/toil.egg-info/SOURCES.txt' reading manifest file 'src/toil.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' writing manifest file 'src/toil.egg-info/SOURCES.txt' running build_ext Creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/.local/lib/python3.12/site-packages/toil.egg-link (link to src) Adding toil 6.1.0 to easy-install.pth file Installing _toil_contained_executor script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/.local/bin Installing _toil_mesos_executor script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/.local/bin Installing _toil_worker script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/.local/bin Installing cwltoil script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/.local/bin Installing toil script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/.local/bin Installing toil-cwl-runner script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/.local/bin Installing toil-wdl-runner script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/.local/bin Installing toil-wes-cwl-runner script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/.local/bin Installed /build/reproducible-path/toil-6.1.0/src Processing dependencies for toil==6.1.0 Searching for urllib3==1.26.18 Best match: urllib3 1.26.18 Adding urllib3 1.26.18 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for typing-extensions==4.10.0 Best match: typing-extensions 4.10.0 Adding typing-extensions 4.10.0 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for ruamel.yaml==0.18.6 Best match: ruamel.yaml 0.18.6 Adding ruamel.yaml 0.18.6 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for requests==2.31.0 Best match: requests 2.31.0 Adding requests 2.31.0 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for PyYAML==6.0.1 Best match: PyYAML 6.0.1 Adding PyYAML 6.0.1 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for pytz==2024.1 Best match: pytz 2024.1 Adding pytz 2024.1 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for python-dateutil==2.9.0 Best match: python-dateutil 2.9.0 Adding python-dateutil 2.9.0 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for psutil==5.9.8 Best match: psutil 5.9.8 Adding psutil 5.9.8 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for enlighten==1.11.2 Best match: enlighten 1.11.2 Adding enlighten 1.11.2 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for docker==6.1.3 Best match: docker 6.1.3 Adding docker 6.1.3 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for dill==0.3.8 Best match: dill 0.3.8 Adding dill 0.3.8 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for ConfigArgParse==1.7 Best match: ConfigArgParse 1.7 Adding ConfigArgParse 1.7 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for Pypubsub==4.0.3 Best match: Pypubsub 4.0.3 Adding Pypubsub 4.0.3 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for ruamel.yaml.clib==0.2.8 Best match: ruamel.yaml.clib 0.2.8 Adding ruamel.yaml.clib 0.2.8 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for six==1.16.0 Best match: six 1.16.0 Adding six 1.16.0 to easy-install.pth file Using /usr/lib/python3/dist-packages Finished processing dependencies for toil==6.1.0 ============================= test session starts ============================== platform linux -- Python 3.12.3, pytest-8.1.2, pluggy-1.5.0 -- /usr/bin/python3.12 cachedir: .pytest_cache rootdir: /build/reproducible-path/toil-6.1.0 configfile: setup.cfg plugins: xdist-3.5.0 created: 4/4 workers 4 workers [515 items] scheduling tests via LoadScopeScheduling src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testBatchCreate src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testBatchCreate src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testBatchCreate src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testBatchCreate [gw0] [ 0%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testBatchCreate src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testChildLoadingEquality [gw0] [ 0%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testChildLoadingEquality src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testCleanCache [gw2] [ 0%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testBatchCreate [gw3] [ 0%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testBatchCreate [gw0] [ 0%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testCleanCache src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testConfigEquality src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testChildLoadingEquality src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testChildLoadingEquality [gw0] [ 1%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testConfigEquality src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testDestructionIdempotence [gw2] [ 1%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testChildLoadingEquality [gw3] [ 1%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testChildLoadingEquality [gw0] [ 1%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testDestructionIdempotence src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testDestructionOfCorruptedJobStore src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testCleanCache [gw0] [ 1%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testDestructionOfCorruptedJobStore src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testEmptyFileStoreIDIsReadable src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testCleanCache [gw2] [ 2%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testCleanCache [gw0] [ 2%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testEmptyFileStoreIDIsReadable src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testEncrypted [gw3] [ 2%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testCleanCache src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testConfigEquality src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testConfigEquality [gw0] [ 2%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testEncrypted src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testFileDeletion [gw0] [ 2%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testFileDeletion [gw3] [ 3%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testConfigEquality [gw2] [ 3%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testConfigEquality src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testDestructionIdempotence src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testGrowingAndShrinkingJob [gw0] [ 3%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testGrowingAndShrinkingJob src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testDestructionIdempotence [gw2] [ 3%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testDestructionIdempotence [gw3] [ 3%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testDestructionIdempotence src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testImportHttpFile [gw0] [ 4%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testImportHttpFile src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testInitialState src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testDestructionOfCorruptedJobStore [gw0] [ 4%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testInitialState src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testDestructionOfCorruptedJobStore [gw3] [ 4%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testDestructionOfCorruptedJobStore src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testInlinedFiles [gw0] [ 4%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testInlinedFiles [gw2] [ 4%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testDestructionOfCorruptedJobStore src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testEmptyFileStoreIDIsReadable src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testEmptyFileStoreIDIsReadable src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testJobCreation [gw0] [ 5%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testJobCreation src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testJobDeletions [gw0] [ 5%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testJobDeletions [gw2] [ 5%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testEmptyFileStoreIDIsReadable [gw3] [ 5%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testEmptyFileStoreIDIsReadable src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testFileDeletion src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testFileDeletion src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testJobLoadEquality [gw3] [ 5%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testFileDeletion [gw2] [ 6%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testFileDeletion [gw0] [ 6%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testJobLoadEquality src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testLargeFile [gw0] [ 6%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testLargeFile src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testMultiThreadImportFile [gw0] [ 6%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testMultiThreadImportFile src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testMultipartUploads src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testGrowingAndShrinkingJob [gw0] [ 6%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testMultipartUploads src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testOverlargeJob [gw3] [ 6%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testGrowingAndShrinkingJob src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testImportHttpFile [gw0] [ 7%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testOverlargeJob src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testPartialReadFromStream src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testGrowingAndShrinkingJob [gw2] [ 7%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testGrowingAndShrinkingJob [gw3] [ 7%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testImportHttpFile src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testInitialState [gw0] [ 7%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testPartialReadFromStream src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testPerJobFiles [gw0] [ 7%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testPerJobFiles src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testImportHttpFile [gw3] [ 8%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testInitialState [gw2] [ 8%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testImportHttpFile src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testPersistantFilesToDelete [gw0] [ 8%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testPersistantFilesToDelete src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testReadWriteFileStreamTextMode src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testJobCreation src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testInitialState [gw0] [ 8%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testReadWriteFileStreamTextMode src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testReadWriteSharedFilesTextMode [gw2] [ 8%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testInitialState [gw0] [ 9%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testReadWriteSharedFilesTextMode src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testInlinedFiles src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testSDBDomainsDeletedOnFailedJobstoreBucketCreation [gw0] [ 9%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testSDBDomainsDeletedOnFailedJobstoreBucketCreation [gw2] [ 9%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testInlinedFiles src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testSharedFiles src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testJobCreation [gw0] [ 9%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testSharedFiles [gw3] [ 9%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testJobCreation src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testJobDeletions src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testStatsAndLogging [gw0] [ 10%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testStatsAndLogging [gw2] [ 10%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testJobCreation [gw3] [ 10%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testJobDeletions src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testJobLoadEquality [gw3] [ 10%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testJobLoadEquality src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testJobDeletions src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testUpdateBehavior [gw0] [ 10%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testUpdateBehavior src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testWriteLogFiles [gw2] [ 11%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testJobDeletions [gw0] [ 11%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testWriteLogFiles src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testZeroLengthFiles src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testLargeFile [gw3] [ 11%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testLargeFile [gw0] [ 11%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testZeroLengthFiles src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testJobLoadEquality [gw2] [ 11%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testJobLoadEquality [gw0] [ 12%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testLargeFile [gw0] [ 12%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testMultipartUploads [gw3] [ 12%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testMultipartUploads [gw2] [ 12%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testLargeFile src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testMultiThreadImportFile [gw2] [ 12%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testMultiThreadImportFile src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testPartialReadFromStream [gw3] [ 13%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testPartialReadFromStream [gw0] [ 13%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testMultipartUploads src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testPerJobFiles [gw2] [ 13%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testMultipartUploads [gw3] [ 13%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testPerJobFiles src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testOverlargeJob src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testPersistantFilesToDelete [gw3] [ 13%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testPersistantFilesToDelete [gw0] [ 13%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw2] [ 14%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testOverlargeJob src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testPartialReadFromStream src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw0] [ 14%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw2] [ 14%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testPartialReadFromStream src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testReadWriteFileStreamTextMode [gw0] [ 14%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testPerJobFiles [gw0] [ 14%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw2] [ 15%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testPerJobFiles src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testPersistantFilesToDelete [gw0] [ 15%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw3] [ 15%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testReadWriteFileStreamTextMode src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testReadWriteSharedFilesTextMode [gw3] [ 15%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testReadWriteSharedFilesTextMode src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testSharedFiles [gw0] [ 15%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw2] [ 16%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testPersistantFilesToDelete src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testReadWriteFileStreamTextMode [gw3] [ 16%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testSharedFiles [gw0] [ 16%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw0] [ 16%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw2] [ 16%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testReadWriteFileStreamTextMode src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw0] [ 17%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testReadWriteSharedFilesTextMode src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testStatsAndLogging [gw3] [ 17%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testStatsAndLogging src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testUpdateBehavior [gw2] [ 17%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testReadWriteSharedFilesTextMode src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testSDBDomainsDeletedOnFailedJobstoreBucketCreation src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py [gw0] [ 17%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py [gw3] [ 17%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testUpdateBehavior src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testWriteLogFiles [gw2] [ 18%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testSDBDomainsDeletedOnFailedJobstoreBucketCreation src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean [gw3] [ 18%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testWriteLogFiles src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testZeroLengthFiles [gw0] [ 18%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_FORGO [gw0] [ 18%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_FORGO src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testSharedFiles [gw2] [ 18%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testSharedFiles src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testStatsAndLogging src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_None [gw0] [ 19%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_None src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_RM [gw2] [ 19%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testStatsAndLogging [gw0] [ 19%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_RM src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_STOP src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testUpdateBehavior [gw0] [ 19%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_STOP src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_FORGO [gw0] [ 19%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_FORGO [gw2] [ 20%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testUpdateBehavior src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_None src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testWriteLogFiles [gw0] [ 20%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_None src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_RM [gw3] [ 20%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testZeroLengthFiles src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw3] [ 20%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw0] [ 20%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_RM [gw2] [ 20%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testWriteLogFiles src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testZeroLengthFiles src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_STOP [gw0] [ 21%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_STOP src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_FORGO [gw2] [ 21%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testZeroLengthFiles [gw0] [ 21%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_FORGO src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_None src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw3] [ 21%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw0] [ 21%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_None src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_RM [gw2] [ 22%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw0] [ 22%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_RM src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_STOP src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw0] [ 22%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_STOP src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_FORGO [gw0] [ 22%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_FORGO src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py [gw3] [ 22%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_None [gw2] [ 23%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py [gw0] [ 23%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_None src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_RM [gw0] [ 23%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_RM src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_STOP [gw2] [ 23%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py [gw3] [ 23%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw3] [ 24%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw3] [ 24%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw3] [ 24%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw3] [ 24%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py [gw3] [ 24%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py [gw0] [ 25%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_STOP src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_FORGO [gw0] [ 25%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_FORGO src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_None [gw0] [ 25%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_None [gw2] [ 25%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw3] [ 25%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw3] [ 26%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_RM [gw0] [ 26%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_RM src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_STOP [gw0] [ 26%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_STOP src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_FORGO [gw0] [ 26%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_FORGO [gw2] [ 26%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_None [gw3] [ 26%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw0] [ 27%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_None [gw2] [ 27%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_RM [gw0] [ 27%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_RM src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_STOP [gw0] [ 27%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_STOP src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs [gw0] [ 27%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs [gw2] [ 28%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw2] [ 28%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs_Demux [gw0] [ 28%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs_Demux src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs_Demux_Stream [gw0] [ 28%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs_Demux_Stream src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs_Stream [gw0] [ 28%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs_Stream [gw2] [ 29%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerPipeChain [gw0] [ 29%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerPipeChain src/toil/test/lib/dockerTest.py::DockerTest::testDockerPipeChainErrorDetection [gw0] [ 29%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerPipeChainErrorDetection src/toil/test/lib/dockerTest.py::DockerTest::testNonCachingDockerChain [gw0] [ 29%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testNonCachingDockerChain src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py [gw3] [ 29%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testAsyncWriteWithCaching [gw3] [ 30%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testAsyncWriteWithCaching src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testCacheEvictionFailCase [gw3] [ 30%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testCacheEvictionFailCase [gw2] [ 30%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testCacheEvictionPartialEvict [gw3] [ 30%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testCacheEvictionPartialEvict src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testCacheEvictionTotalEvict [gw3] [ 30%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testCacheEvictionTotalEvict src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testControlledFailedWorkerRetry src/toil/test/lib/dockerTest.py::DockerTest::testNonCachingDockerChainErrorDetection [gw0] [ 31%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testNonCachingDockerChainErrorDetection src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testAsyncWriteWithCaching [gw0] [ 31%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testAsyncWriteWithCaching src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testCacheEvictionFailCase [gw3] [ 31%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testControlledFailedWorkerRetry src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testDeleteLocalFile src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw2] [ 31%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw2] [ 31%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw0] [ 32%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testCacheEvictionFailCase src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testCacheEvictionPartialEvict [gw0] [ 32%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testCacheEvictionPartialEvict src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testCacheEvictionTotalEvict [gw0] [ 32%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testCacheEvictionTotalEvict src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testControlledFailedWorkerRetry [gw2] [ 32%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py [gw0] [ 32%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testControlledFailedWorkerRetry src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testDeleteLocalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testAsyncWriteWithCaching [gw0] [ 33%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testDeleteLocalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testExtremeCacheSetup [gw2] [ 33%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testAsyncWriteWithCaching src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testCacheEvictionFailCase [gw0] [ 33%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testExtremeCacheSetup [gw2] [ 33%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testCacheEvictionFailCase src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testFileStoreExportFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testCacheEvictionPartialEvict [gw0] [ 33%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testFileStoreExportFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testFileStoreLogging [gw2] [ 33%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testCacheEvictionPartialEvict src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testCacheEvictionTotalEvict [gw0] [ 34%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testFileStoreLogging src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testFileStoreOperations [gw2] [ 34%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testCacheEvictionTotalEvict src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testControlledFailedWorkerRetry [gw0] [ 34%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testFileStoreOperations src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testMultipleJobsReadSameCacheHitGlobalFile [gw2] [ 34%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testControlledFailedWorkerRetry src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testDeleteLocalFile [gw0] [ 34%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testMultipleJobsReadSameCacheHitGlobalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testMultipleJobsReadSameCacheMissGlobalFile [gw2] [ 35%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testDeleteLocalFile [gw0] [ 35%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testMultipleJobsReadSameCacheMissGlobalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testExtremeCacheSetup src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadCachHitFileFromJobStore [gw2] [ 35%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testExtremeCacheSetup [gw0] [ 35%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadCachHitFileFromJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testFileStoreExportFile [gw0] [ 35%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile [gw2] [ 36%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testFileStoreExportFile [gw0] [ 36%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadWriteFileStreamTextMode src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testFileStoreLogging [gw0] [ 36%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadWriteFileStreamTextMode src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testRemoveLocalImmutablyReadFile [gw2] [ 36%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testFileStoreLogging [gw0] [ 36%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testRemoveLocalImmutablyReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testFileStoreOperations src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testRemoveLocalMutablyReadFile [gw0] [ 37%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testRemoveLocalMutablyReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReturnFileSizes [gw2] [ 37%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testFileStoreOperations [gw0] [ 37%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReturnFileSizes [gw1] [ 37%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testBatchCreate src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testChildLoadingEquality src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testMultipleJobsReadSameCacheHitGlobalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReturnFileSizesWithBadWorker [gw0] [ 37%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReturnFileSizesWithBadWorker src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testSimultaneousReadsUncachedStream [gw2] [ 38%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testMultipleJobsReadSameCacheHitGlobalFile [gw0] [ 38%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testSimultaneousReadsUncachedStream src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testMultipleJobsReadSameCacheMissGlobalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testToilIsNotBroken [gw0] [ 38%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testToilIsNotBroken src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteExportFileCompatibility [gw2] [ 38%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testMultipleJobsReadSameCacheMissGlobalFile [gw0] [ 38%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteExportFileCompatibility src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadCachHitFileFromJobStore [gw0] [ 39%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteNonLocalFileToJobStore [gw2] [ 39%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadCachHitFileFromJobStore [gw0] [ 39%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteNonLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteReadGlobalFilePermissions [gw0] [ 39%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteReadGlobalFilePermissions src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_many_all_exist [gw2] [ 39%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile [gw2] [ 40%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile [gw1] [ 40%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testChildLoadingEquality src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadWriteFileStreamTextMode src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testCleanCache [gw2] [ 40%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadWriteFileStreamTextMode [gw1] [ 40%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testCleanCache src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testRemoveLocalImmutablyReadFile src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testConfigEquality [gw2] [ 40%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testRemoveLocalImmutablyReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testRemoveLocalMutablyReadFile [gw2] [ 40%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testRemoveLocalMutablyReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReturnFileSizes [gw2] [ 41%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReturnFileSizes src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReturnFileSizesWithBadWorker [gw2] [ 41%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReturnFileSizesWithBadWorker src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testSimultaneousReadsUncachedStream [gw2] [ 41%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testSimultaneousReadsUncachedStream src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testToilIsNotBroken [gw2] [ 41%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testToilIsNotBroken src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteExportFileCompatibility [gw2] [ 41%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteExportFileCompatibility src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteLocalFileToJobStore [gw2] [ 42%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteNonLocalFileToJobStore [gw2] [ 42%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteNonLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteReadGlobalFilePermissions [gw2] [ 42%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteReadGlobalFilePermissions src/toil/test/utils/utilsTest.py::UtilsTest::testAWSProvisionerUtils [gw1] [ 42%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testConfigEquality [gw2] [ 42%] SKIPPED src/toil/test/utils/utilsTest.py::UtilsTest::testAWSProvisionerUtils src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testDestructionIdempotence src/toil/test/utils/utilsTest.py::UtilsTest::testGetPIDStatus [gw1] [ 43%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testDestructionIdempotence src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testDestructionOfCorruptedJobStore [gw1] [ 43%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testDestructionOfCorruptedJobStore src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testEmptyFileStoreIDIsReadable [gw0] [ 43%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_many_all_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_one_exists [gw1] [ 43%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testEmptyFileStoreIDIsReadable src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testFileDeletion [gw1] [ 43%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testFileDeletion src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testGrowingAndShrinkingJob [gw0] [ 44%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_one_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_one_not_exists [gw0] [ 44%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_one_not_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_sacct_raises_job_exists [gw1] [ 44%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testGrowingAndShrinkingJob src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testImportHttpFile [gw0] [ 44%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_sacct_raises_job_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_sacct_raises_job_not_exists [gw0] [ 44%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_sacct_raises_job_not_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_some_exists [gw0] [ 45%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_some_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_many_all_exist [gw0] [ 45%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_many_all_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_many_none_exist [gw0] [ 45%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_many_none_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_many_some_exist [gw0] [ 45%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_many_some_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_one_exists [gw0] [ 45%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_one_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_one_not_exists [gw0] [ 46%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_one_not_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_many_all_exist [gw1] [ 46%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testImportHttpFile src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testInitialState [gw0] [ 46%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_many_all_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_many_none_exist [gw0] [ 46%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_many_none_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_many_some_exist [gw0] [ 46%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_many_some_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_one_exists [gw0] [ 46%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_one_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_one_not_exists [gw0] [ 47%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_one_not_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_job_exists [gw0] [ 47%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_job_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_job_not_exists [gw0] [ 47%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_job_not_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_sacct_raises_job_exists [gw0] [ 47%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_sacct_raises_job_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_sacct_raises_job_not_exists [gw3] [ 47%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testDeleteLocalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testExtremeCacheSetup [gw0] [ 48%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_sacct_raises_job_not_exists src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testApplianceParser [gw3] [ 48%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testExtremeCacheSetup [gw0] [ 48%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testApplianceParser src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadGoogleRepo src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testFileStoreExportFile [gw0] [ 48%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadGoogleRepo src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayRepo [gw0] [ 48%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayRepo src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayRepoNTag [gw3] [ 49%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testFileStoreExportFile [gw0] [ 49%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayRepoNTag src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayTag [gw0] [ 49%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayTag src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testFileStoreLogging src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBroadDockerRepo [gw0] [ 49%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBroadDockerRepo [gw3] [ 49%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testFileStoreLogging src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testFileStoreOperations src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBroadDockerRepoBadTag [gw3] [ 50%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testFileStoreOperations src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testMultipleJobsReadSameCacheHitGlobalFile [gw0] [ 50%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBroadDockerRepoBadTag [gw3] [ 50%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testMultipleJobsReadSameCacheHitGlobalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testMultipleJobsReadSameCacheMissGlobalFile src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testGoogleRepo [gw0] [ 50%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testGoogleRepo [gw3] [ 50%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testMultipleJobsReadSameCacheMissGlobalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCachHitFileFromJobStore src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testNonexistentRepo [gw0] [ 51%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testNonexistentRepo src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testOfficialUbuntuRepo [gw0] [ 51%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testOfficialUbuntuRepo src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testToilQuayRepo [gw0] [ 51%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testToilQuayRepo src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testBetaInertia [gw0] [ 51%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testBetaInertia src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testMaxNodes [gw0] [ 51%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testMaxNodes src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testMinNodes [gw0] [ 52%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testMinNodes src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testNoLaunchingIfDeltaAlreadyMet [gw0] [ 52%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testNoLaunchingIfDeltaAlreadyMet src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testPreemptibleDeficitIsSet [gw0] [ 52%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testPreemptibleDeficitIsSet src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testPreemptibleDeficitResponse [gw0] [ 52%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testPreemptibleDeficitResponse src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testRounding [gw0] [ 52%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testRounding src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_large [gw0] [ 53%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_large src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_observed [gw0] [ 53%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_observed src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_small [gw0] [ 53%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_small src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_and_cancel_workflows [gw0] [ 53%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_and_cancel_workflows src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_https_url [gw0] [ 53%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_https_url src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_inputs_zip [gw0] [ 53%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_inputs_zip src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_manifest_and_inputs_zip [gw0] [ 54%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_manifest_and_inputs_zip src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_manifest_zip [gw0] [ 54%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_manifest_zip src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_multi_file_zip [gw0] [ 54%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_multi_file_zip src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_no_params_zip [gw0] [ 54%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_no_params_zip src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_relative_url [gw0] [ 54%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_relative_url src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_relative_url_no_attachments_fails [gw0] [ 55%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_relative_url_no_attachments_fails src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_single_file_zip [gw0] [ 55%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_single_file_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_and_cancel_workflows [gw0] [ 55%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_and_cancel_workflows src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_https_url [gw0] [ 55%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_https_url src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_inputs_zip [gw0] [ 55%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_inputs_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_manifest_and_inputs_zip [gw0] [ 56%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_manifest_and_inputs_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_manifest_zip [gw0] [ 56%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_manifest_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_multi_file_zip [gw0] [ 56%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_multi_file_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_no_params_zip [gw0] [ 56%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_no_params_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_relative_url [gw0] [ 56%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_relative_url src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_relative_url_no_attachments_fails [gw0] [ 57%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_relative_url_no_attachments_fails src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_single_file_zip [gw0] [ 57%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_single_file_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_and_cancel_workflows [gw0] [ 57%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_and_cancel_workflows src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_https_url [gw0] [ 57%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_https_url src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_inputs_zip [gw0] [ 57%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_inputs_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_manifest_and_inputs_zip [gw0] [ 58%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_manifest_and_inputs_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_manifest_zip [gw0] [ 58%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_manifest_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_multi_file_zip [gw0] [ 58%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_multi_file_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_no_params_zip [gw0] [ 58%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_no_params_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_relative_url [gw0] [ 58%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_relative_url src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_relative_url_no_attachments_fails [gw0] [ 59%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_relative_url_no_attachments_fails src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_single_file_zip [gw0] [ 59%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_single_file_zip src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testAddingInitialNode [gw0] [ 59%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testAddingInitialNode src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testHighTargetTime [gw3] [ 59%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCachHitFileFromJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile [gw0] [ 59%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testHighTargetTime src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testJobTooLargeForAllNodes [gw0] [ 60%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testJobTooLargeForAllNodes src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testLongRunningJobs [gw3] [ 60%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile [gw0] [ 60%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testLongRunningJobs src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testLowTargetTime [gw0] [ 60%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testLowTargetTime src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testPackingOneShape [gw0] [ 60%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testPackingOneShape src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testPathologicalCase [gw0] [ 60%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testPathologicalCase src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testSorting [gw0] [ 61%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testSorting src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testZeroResourceJobs [gw0] [ 61%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testZeroResourceJobs src/toil/test/src/jobTest.py::JobTest::testDAGConsistency [gw3] [ 61%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode [gw0] [ 61%] FAILED src/toil/test/src/jobTest.py::JobTest::testDAGConsistency src/toil/test/src/jobTest.py::JobTest::testDeadlockDetection [gw3] [ 61%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testRemoveLocalImmutablyReadFile [gw3] [ 62%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testRemoveLocalImmutablyReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testRemoveLocalMutablyReadFile [gw3] [ 62%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testRemoveLocalMutablyReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReturnFileSizes [gw3] [ 62%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReturnFileSizes src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReturnFileSizesWithBadWorker [gw3] [ 62%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReturnFileSizesWithBadWorker src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testSimultaneousReadsUncachedStream [gw2] [ 62%] PASSED src/toil/test/utils/utilsTest.py::UtilsTest::testGetPIDStatus src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedCWLWF [gw2] [ 63%] SKIPPED src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedCWLWF src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedToilWF [gw3] [ 63%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testSimultaneousReadsUncachedStream src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testToilIsNotBroken [gw0] [ 63%] PASSED src/toil/test/src/jobTest.py::JobTest::testDeadlockDetection src/toil/test/src/jobTest.py::JobTest::testEvaluatingRandomDAG [gw0] [ 63%] SKIPPED src/toil/test/src/jobTest.py::JobTest::testEvaluatingRandomDAG src/toil/test/src/jobTest.py::JobTest::testNewCheckpointIsLeafVertexNonRootCase [gw0] [ 63%] SKIPPED src/toil/test/src/jobTest.py::JobTest::testNewCheckpointIsLeafVertexNonRootCase src/toil/test/src/jobTest.py::JobTest::testNewCheckpointIsLeafVertexRootCase [gw0] [ 64%] SKIPPED src/toil/test/src/jobTest.py::JobTest::testNewCheckpointIsLeafVertexRootCase src/toil/test/src/jobTest.py::JobTest::testSiblingDAGConsistency [gw0] [ 64%] SKIPPED src/toil/test/src/jobTest.py::JobTest::testSiblingDAGConsistency src/toil/test/src/jobTest.py::JobTest::testStatic [gw0] [ 64%] SKIPPED src/toil/test/src/jobTest.py::JobTest::testStatic src/toil/test/src/jobTest.py::JobTest::testStatic2 [gw3] [ 64%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testToilIsNotBroken src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility [gw0] [ 64%] FAILED src/toil/test/src/jobTest.py::JobTest::testStatic2 src/toil/test/src/jobTest.py::JobTest::testTrivialDAGConsistency [gw0] [ 65%] SKIPPED src/toil/test/src/jobTest.py::JobTest::testTrivialDAGConsistency src/toil/test/src/miscTests.py::MiscTests::testGetSizeOfDirectoryWorks [gw0] [ 65%] SKIPPED src/toil/test/src/miscTests.py::MiscTests::testGetSizeOfDirectoryWorks src/toil/test/src/miscTests.py::MiscTests::testIDStability [gw0] [ 65%] PASSED src/toil/test/src/miscTests.py::MiscTests::testIDStability src/toil/test/src/miscTests.py::MiscTests::test_atomic_context_error [gw0] [ 65%] PASSED src/toil/test/src/miscTests.py::MiscTests::test_atomic_context_error src/toil/test/src/miscTests.py::MiscTests::test_atomic_context_ok [gw0] [ 65%] PASSED src/toil/test/src/miscTests.py::MiscTests::test_atomic_context_ok src/toil/test/src/miscTests.py::MiscTests::test_atomic_install [gw0] [ 66%] PASSED src/toil/test/src/miscTests.py::MiscTests::test_atomic_install src/toil/test/src/miscTests.py::MiscTests::test_atomic_install_dev [gw0] [ 66%] PASSED src/toil/test/src/miscTests.py::MiscTests::test_atomic_install_dev src/toil/test/src/miscTests.py::MiscTests::test_call_command_err [gw0] [ 66%] PASSED src/toil/test/src/miscTests.py::MiscTests::test_call_command_err src/toil/test/src/miscTests.py::MiscTests::test_call_command_ok [gw0] [ 66%] PASSED src/toil/test/src/miscTests.py::MiscTests::test_call_command_ok src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testFileStoreLogging [gw0] [ 66%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testFileStoreLogging src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testFileStoreOperations [gw0] [ 66%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testFileStoreOperations src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode [gw3] [ 67%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteLocalFileToJobStore [gw0] [ 67%] FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testToilIsNotBroken [gw0] [ 67%] FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testToilIsNotBroken src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility [gw3] [ 67%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteNonLocalFileToJobStore [gw1] [ 67%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testInitialState src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testJobCreation [gw1] [ 68%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testJobCreation src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testJobDeletions [gw0] [ 68%] FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions [gw3] [ 68%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteNonLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions [gw0] [ 68%] FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testFileStoreLogging [gw0] [ 68%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testFileStoreLogging src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testFileStoreOperations [gw0] [ 69%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testFileStoreOperations src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testReadWriteFileStreamTextMode [gw0] [ 69%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testReadWriteFileStreamTextMode src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testToilIsNotBroken [gw0] [ 69%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testToilIsNotBroken src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testWriteExportFileCompatibility [gw0] [ 69%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testWriteExportFileCompatibility src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testWriteReadGlobalFilePermissions [gw0] [ 69%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testWriteReadGlobalFilePermissions src/toil/test/src/jobServiceTest.py::JobServiceTest::testService [gw0] [ 70%] SKIPPED src/toil/test/src/jobServiceTest.py::JobServiceTest::testService src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceDeadlock [gw0] [ 70%] SKIPPED src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceDeadlock src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceParallelRecursive [gw0] [ 70%] SKIPPED src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceParallelRecursive src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceRecursive [gw0] [ 70%] SKIPPED src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceRecursive src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceSerialization [gw0] [ 70%] SKIPPED src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceSerialization src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceWithCheckpoints [gw0] [ 71%] SKIPPED src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceWithCheckpoints src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testService [gw0] [ 71%] SKIPPED src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testService src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceDeadlock [gw0] [ 71%] SKIPPED src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceDeadlock src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceParallelRecursive [gw0] [ 71%] SKIPPED src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceParallelRecursive src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceRecursive [gw0] [ 71%] SKIPPED src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceRecursive src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceSerialization [gw0] [ 72%] SKIPPED src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceSerialization src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceWithCheckpoints [gw0] [ 72%] SKIPPED src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceWithCheckpoints src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_build_tag [gw0] [ 72%] PASSED src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_build_tag src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_build_tag_with_tags [gw0] [ 72%] PASSED src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_build_tag_with_tags src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_empty_aws_tags [gw0] [ 72%] PASSED src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_empty_aws_tags src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_incorrect_json_emoji [gw0] [ 73%] PASSED src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_incorrect_json_emoji src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_incorrect_json_object [gw0] [ 73%] PASSED src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_incorrect_json_object src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testDeferralWithConcurrentEncapsulation [gw0] [ 73%] SKIPPED src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testDeferralWithConcurrentEncapsulation src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testDeferralWithFailureAndEncapsulation [gw0] [ 73%] SKIPPED src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testDeferralWithFailureAndEncapsulation src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testRestart [gw0] [ 73%] SKIPPED src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testRestart src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testSplitRootPackages [gw0] [ 73%] SKIPPED src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testSplitRootPackages src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testUserTypesInJobFunctionArgs [gw0] [ 74%] SKIPPED src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testUserTypesInJobFunctionArgs src/toil/test/src/regularLogTest.py::RegularLogTest::testLogToMaster [gw0] [ 74%] SKIPPED src/toil/test/src/regularLogTest.py::RegularLogTest::testLogToMaster src/toil/test/src/regularLogTest.py::RegularLogTest::testMultipleLogToMaster [gw0] [ 74%] SKIPPED src/toil/test/src/regularLogTest.py::RegularLogTest::testMultipleLogToMaster src/toil/test/src/regularLogTest.py::RegularLogTest::testRegularLog [gw3] [ 74%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testFileStoreLogging [gw3] [ 74%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testFileStoreLogging src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testFileStoreOperations [gw3] [ 75%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testFileStoreOperations src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testReadWriteFileStreamTextMode [gw3] [ 75%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testReadWriteFileStreamTextMode src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testToilIsNotBroken [gw3] [ 75%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testToilIsNotBroken src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testWriteExportFileCompatibility [gw3] [ 75%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testWriteExportFileCompatibility src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testWriteReadGlobalFilePermissions [gw3] [ 75%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testWriteReadGlobalFilePermissions src/toil/test/src/resourceTest.py::ResourceTest::testBuiltIn [gw3] [ 76%] PASSED src/toil/test/src/resourceTest.py::ResourceTest::testBuiltIn src/toil/test/src/resourceTest.py::ResourceTest::testNonPyStandAlone [gw3] [ 76%] PASSED src/toil/test/src/resourceTest.py::ResourceTest::testNonPyStandAlone src/toil/test/src/resourceTest.py::ResourceTest::testPackage [gw3] [ 76%] PASSED src/toil/test/src/resourceTest.py::ResourceTest::testPackage src/toil/test/src/resourceTest.py::ResourceTest::testStandAlone [gw3] [ 76%] PASSED src/toil/test/src/resourceTest.py::ResourceTest::testStandAlone src/toil/test/src/resourceTest.py::ResourceTest::testStandAloneInPackage [gw3] [ 76%] PASSED src/toil/test/src/resourceTest.py::ResourceTest::testStandAloneInPackage src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testCachingFileStore [gw3] [ 77%] FAILED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testCachingFileStore src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStore [gw0] [ 77%] PASSED src/toil/test/src/regularLogTest.py::RegularLogTest::testRegularLog src/toil/test/src/regularLogTest.py::RegularLogTest::testWriteGzipLogs [gw0] [ 77%] SKIPPED src/toil/test/src/regularLogTest.py::RegularLogTest::testWriteGzipLogs src/toil/test/src/regularLogTest.py::RegularLogTest::testWriteLogs [gw2] [ 77%] FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedToilWF src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusSuccessfulCWLWF [gw2] [ 77%] SKIPPED src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusSuccessfulCWLWF src/toil/test/utils/utilsTest.py::UtilsTest::testMultipleJobsPerWorkerStats [gw2] [ 78%] SKIPPED src/toil/test/utils/utilsTest.py::UtilsTest::testMultipleJobsPerWorkerStats src/toil/test/utils/utilsTest.py::UtilsTest::testPrintJobLog [gw0] [ 78%] PASSED src/toil/test/src/regularLogTest.py::RegularLogTest::testWriteLogs src/toil/test/src/miscTests.py::TestPanic::test_nested_panic [gw0] [ 78%] PASSED src/toil/test/src/miscTests.py::TestPanic::test_nested_panic src/toil/test/src/miscTests.py::TestPanic::test_panic [gw0] [ 78%] PASSED src/toil/test/src/miscTests.py::TestPanic::test_panic src/toil/test/src/miscTests.py::TestPanic::test_panic_by_hand [gw0] [ 78%] PASSED src/toil/test/src/miscTests.py::TestPanic::test_panic_by_hand src/toil/test/src/miscTests.py::TestPanic::test_panic_with_secondary [gw0] [ 79%] PASSED src/toil/test/src/miscTests.py::TestPanic::test_panic_with_secondary src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobClass [gw0] [ 79%] SKIPPED src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobClass src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobClassFromMain [gw0] [ 79%] PASSED src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobClassFromMain src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunction [gw2] [ 79%] FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testPrintJobLog src/toil/test/utils/utilsTest.py::UtilsTest::testRestartAttribute [gw0] [ 79%] FAILED src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunction src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunctionFromMain [gw3] [ 80%] FAILED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStore src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStoreWithBadWorker [gw3] [ 80%] SKIPPED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStoreWithBadWorker src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testNonCachingFileStore [gw0] [ 80%] PASSED src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunctionFromMain src/toil/test/lib/test_conversions.py::ConversionTest::test_convert [gw0] [ 80%] PASSED src/toil/test/lib/test_conversions.py::ConversionTest::test_convert src/toil/test/lib/test_conversions.py::ConversionTest::test_hms_duration_to_seconds [gw0] [ 80%] PASSED src/toil/test/lib/test_conversions.py::ConversionTest::test_hms_duration_to_seconds src/toil/test/lib/test_conversions.py::ConversionTest::test_human2bytes [gw0] [ 80%] PASSED src/toil/test/lib/test_conversions.py::ConversionTest::test_human2bytes src/toil/test/server/serverTest.py::ToilWESServerBenchTest::test_get_service_info [gw0] [ 81%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerBenchTest::test_get_service_info src/toil/test/server/serverTest.py::ToilWESServerBenchTest::test_health [gw0] [ 81%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerBenchTest::test_health src/toil/test/server/serverTest.py::ToilWESServerBenchTest::test_home [gw0] [ 81%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerBenchTest::test_home src/toil/test/src/busTest.py::MessageBusTest::test_cross_thread_messaging [gw0] [ 81%] PASSED src/toil/test/src/busTest.py::MessageBusTest::test_cross_thread_messaging src/toil/test/src/busTest.py::MessageBusTest::test_enum_ints_in_file [gw0] [ 81%] PASSED src/toil/test/src/busTest.py::MessageBusTest::test_enum_ints_in_file src/toil/test/src/busTest.py::MessageBusTest::test_restart_without_bus_path [gw3] [ 82%] FAILED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testNonCachingFileStore src/toil/test/provisioners/clusterScalerTest.py::ScalerThreadTest::testClusterScaling [gw3] [ 82%] SKIPPED src/toil/test/provisioners/clusterScalerTest.py::ScalerThreadTest::testClusterScaling src/toil/test/provisioners/clusterScalerTest.py::ScalerThreadTest::testClusterScalingMultipleNodeTypes [gw3] [ 82%] SKIPPED src/toil/test/provisioners/clusterScalerTest.py::ScalerThreadTest::testClusterScalingMultipleNodeTypes src/toil/test/provisioners/clusterScalerTest.py::ScalerThreadTest::testClusterScalingWithPreemptibleJobs [gw3] [ 82%] SKIPPED src/toil/test/provisioners/clusterScalerTest.py::ScalerThreadTest::testClusterScalingWithPreemptibleJobs src/toil/test/src/importExportFileTest.py::ImportExportFileTest::test_basic_import_export [gw3] [ 82%] PASSED src/toil/test/src/importExportFileTest.py::ImportExportFileTest::test_basic_import_export src/toil/test/src/importExportFileTest.py::ImportExportFileTest::test_import_export_restart_false [gw0] [ 83%] FAILED src/toil/test/src/busTest.py::MessageBusTest::test_restart_without_bus_path src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointNotRetried [gw0] [ 83%] FAILED src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointNotRetried src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointRetriedOnce [gw0] [ 83%] SKIPPED src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointRetriedOnce src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointedRestartSucceeds [gw0] [ 83%] SKIPPED src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointedRestartSucceeds src/toil/test/__init__.py::toil.test.make_tests [gw0] [ 83%] PASSED src/toil/test/__init__.py::toil.test.make_tests src/toil/test/__init__.py::toil.test.timeLimit [gw0] [ 84%] PASSED src/toil/test/__init__.py::toil.test.timeLimit src/toil/test/options/options.py::OptionsTest::test_caching_option_priority [gw0] [ 84%] PASSED src/toil/test/options/options.py::OptionsTest::test_caching_option_priority src/toil/test/options/options.py::OptionsTest::test_default_caching_slurm [gw0] [ 84%] PASSED src/toil/test/options/options.py::OptionsTest::test_default_caching_slurm src/toil/test/provisioners/gceProvisionerTest.py::GCEAutoscaleTest::testAutoScale [gw0] [ 84%] SKIPPED src/toil/test/provisioners/gceProvisionerTest.py::GCEAutoscaleTest::testAutoScale src/toil/test/provisioners/gceProvisionerTest.py::GCEAutoscaleTest::testSpotAutoScale [gw0] [ 84%] SKIPPED src/toil/test/provisioners/gceProvisionerTest.py::GCEAutoscaleTest::testSpotAutoScale src/toil/test/provisioners/gceProvisionerTest.py::GCEStaticAutoscaleTest::testAutoScale [gw0] [ 85%] SKIPPED src/toil/test/provisioners/gceProvisionerTest.py::GCEStaticAutoscaleTest::testAutoScale src/toil/test/provisioners/gceProvisionerTest.py::GCEStaticAutoscaleTest::testSpotAutoScale [gw0] [ 85%] SKIPPED src/toil/test/provisioners/gceProvisionerTest.py::GCEStaticAutoscaleTest::testSpotAutoScale src/toil/test/server/serverTest.py::AWSStateStoreTest::test_state_store [gw0] [ 85%] SKIPPED src/toil/test/server/serverTest.py::AWSStateStoreTest::test_state_store src/toil/test/server/serverTest.py::AWSStateStoreTest::test_state_store_paths [gw0] [ 85%] SKIPPED src/toil/test/server/serverTest.py::AWSStateStoreTest::test_state_store_paths src/toil/test/src/jobDescriptionTest.py::JobDescriptionTest::testJobDescription [gw0] [ 85%] PASSED src/toil/test/src/jobDescriptionTest.py::JobDescriptionTest::testJobDescription src/toil/test/src/jobDescriptionTest.py::JobDescriptionTest::testJobDescriptionSequencing [gw0] [ 86%] PASSED src/toil/test/src/jobDescriptionTest.py::JobDescriptionTest::testJobDescriptionSequencing src/toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testAddChildEncapsulate [gw0] [ 86%] PASSED src/toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testAddChildEncapsulate src/toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testEncapsulation [gw3] [ 86%] PASSED src/toil/test/src/importExportFileTest.py::ImportExportFileTest::test_import_export_restart_false src/toil/test/src/importExportFileTest.py::ImportExportFileTest::test_import_export_restart_true [gw3] [ 86%] SKIPPED src/toil/test/src/importExportFileTest.py::ImportExportFileTest::test_import_export_restart_true src/toil/test/src/toilContextManagerTest.py::ToilContextManagerTest::testContextManger [gw3] [ 86%] SKIPPED src/toil/test/src/toilContextManagerTest.py::ToilContextManagerTest::testContextManger src/toil/test/src/toilContextManagerTest.py::ToilContextManagerTest::testExportAfterFailedExport [gw3] [ 86%] SKIPPED src/toil/test/src/toilContextManagerTest.py::ToilContextManagerTest::testExportAfterFailedExport src/toil/test/src/toilContextManagerTest.py::ToilContextManagerTest::testNoContextManger [gw3] [ 87%] SKIPPED src/toil/test/src/toilContextManagerTest.py::ToilContextManagerTest::testNoContextManger src/toil/test/src/resumabilityTest.py::ResumabilityTest::test [gw3] [ 87%] SKIPPED src/toil/test/src/resumabilityTest.py::ResumabilityTest::test src/toil/test/src/resumabilityTest.py::ResumabilityTest::test_chaining [gw0] [ 87%] FAILED src/toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testEncapsulation src/toil/test/src/restartDAGTest.py::RestartDAGTest::testRestartedWorkflowSchedulesCorrectJobsOnFailedParent [gw0] [ 87%] SKIPPED src/toil/test/src/restartDAGTest.py::RestartDAGTest::testRestartedWorkflowSchedulesCorrectJobsOnFailedParent src/toil/test/src/restartDAGTest.py::RestartDAGTest::testRestartedWorkflowSchedulesCorrectJobsOnKilledParent [gw0] [ 87%] SKIPPED src/toil/test/src/restartDAGTest.py::RestartDAGTest::testRestartedWorkflowSchedulesCorrectJobsOnKilledParent src/toil/test/utils/toilDebugTest.py::testJobStoreContents [gw0] [ 88%] SKIPPED src/toil/test/utils/toilDebugTest.py::testJobStoreContents src/toil/test/utils/toilDebugTest.py::testFetchJobStoreFiles [gw3] [ 88%] FAILED src/toil/test/src/resumabilityTest.py::ResumabilityTest::test_chaining src/toil/test/src/threadingTest.py::ThreadingTest::testGlobalMutexOrdering [gw2] [ 88%] FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testRestartAttribute src/toil/test/utils/utilsTest.py::UtilsTest::testUnicodeSupport [gw0] [ 88%] FAILED src/toil/test/utils/toilDebugTest.py::testFetchJobStoreFiles src/toil/test/utils/toilDebugTest.py::DebugJobTest::test_print_job_info [gw2] [ 88%] FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testUnicodeSupport src/toil/test/utils/utilsTest.py::UtilsTest::testUtilsSort [gw2] [ 89%] SKIPPED src/toil/test/utils/utilsTest.py::UtilsTest::testUtilsSort src/toil/test/utils/utilsTest.py::UtilsTest::testUtilsStatsSort [gw2] [ 89%] SKIPPED src/toil/test/utils/utilsTest.py::UtilsTest::testUtilsStatsSort src/toil/test/utils/utilsTest.py::UtilsTest::test_config_functionality [gw1] [ 89%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testJobDeletions src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testJobLoadEquality [gw1] [ 89%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testJobLoadEquality src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testLargeFile [gw1] [ 89%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testLargeFile src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testMultipartUploads [gw1] [ 90%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testMultipartUploads src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPartialReadFromStream [gw1] [ 90%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPartialReadFromStream src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPerJobFiles [gw1] [ 90%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPerJobFiles src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPersistantFilesToDelete [gw1] [ 90%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPersistantFilesToDelete src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPreserveFileName [gw1] [ 90%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPreserveFileName src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testReadWriteFileStreamTextMode [gw1] [ 91%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testReadWriteFileStreamTextMode src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testReadWriteSharedFilesTextMode [gw1] [ 91%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testReadWriteSharedFilesTextMode src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testSharedFiles [gw1] [ 91%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testSharedFiles src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testStatsAndLogging [gw2] [ 91%] PASSED src/toil/test/utils/utilsTest.py::UtilsTest::test_config_functionality src/toil/test/lib/test_misc.py::UserNameAvailableTest::test_get_user_name [gw2] [ 91%] PASSED src/toil/test/lib/test_misc.py::UserNameAvailableTest::test_get_user_name src/toil/test/lib/test_misc.py::UserNameUnvailableTest::test_get_user_name [gw2] [ 92%] PASSED src/toil/test/lib/test_misc.py::UserNameUnvailableTest::test_get_user_name src/toil/test/lib/test_misc.py::UserNameVeryBrokenTest::test_get_user_name [gw2] [ 92%] PASSED src/toil/test/lib/test_misc.py::UserNameVeryBrokenTest::test_get_user_name src/toil/test/mesos/MesosDataStructuresTest.py::DataStructuresTest::testJobQueue [gw1] [ 92%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testStatsAndLogging src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testUpdateBehavior [gw1] [ 92%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testUpdateBehavior src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testWriteLogFiles [gw2] [ 92%] PASSED src/toil/test/mesos/MesosDataStructuresTest.py::DataStructuresTest::testJobQueue src/toil/test/provisioners/gceProvisionerTest.py::GCEAutoscaleTestMultipleNodeTypes::testAutoScale [gw2] [ 93%] SKIPPED src/toil/test/provisioners/gceProvisionerTest.py::GCEAutoscaleTestMultipleNodeTypes::testAutoScale src/toil/test/provisioners/gceProvisionerTest.py::GCERestartTest::testAutoScaledCluster [gw2] [ 93%] SKIPPED src/toil/test/provisioners/gceProvisionerTest.py::GCERestartTest::testAutoScaledCluster src/toil/test/provisioners/provisionerTest.py::ProvisionerTest::test_node_type_parsing [gw2] [ 93%] PASSED src/toil/test/provisioners/provisionerTest.py::ProvisionerTest::test_node_type_parsing src/toil/test/server/serverTest.py::ToilServerUtilsTest::test_workflow_canceling_recovery [gw2] [ 93%] SKIPPED src/toil/test/server/serverTest.py::ToilServerUtilsTest::test_workflow_canceling_recovery src/toil/test/server/serverTest.py::FileStateStoreTest::test_state_store [gw2] [ 93%] SKIPPED src/toil/test/server/serverTest.py::FileStateStoreTest::test_state_store src/toil/test/server/serverTest.py::FileStateStoreURLTest::test_state_store [gw2] [ 93%] SKIPPED src/toil/test/server/serverTest.py::FileStateStoreURLTest::test_state_store src/toil/test/src/helloWorldTest.py::HelloWorldTest::testHelloWorld [gw1] [ 94%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testWriteLogFiles src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testZeroLengthFiles [gw1] [ 94%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testZeroLengthFiles src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_file_link_imports [gw1] [ 94%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_file_link_imports src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_jobstore_does_not_leak_symlinks [gw1] [ 94%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_jobstore_does_not_leak_symlinks src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_jobstore_init_preserves_symlink_path [gw1] [ 94%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_jobstore_init_preserves_symlink_path src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw1] [ 95%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw1] [ 95%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py [gw1] [ 95%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw1] [ 95%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw1] [ 95%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw1] [ 96%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw1] [ 96%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw1] [ 96%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py [gw1] [ 96%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw1] [ 96%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw1] [ 97%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw1] [ 97%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py [gw0] [ 97%] PASSED src/toil/test/utils/toilDebugTest.py::DebugJobTest::test_print_job_info src/toil/test/utils/toilDebugTest.py::DebugJobTest::test_run_job [gw1] [ 97%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py src/toil/test/src/promisesTest.py::PathIndexingPromiseTest::test [gw1] [ 97%] FAILED src/toil/test/src/promisesTest.py::PathIndexingPromiseTest::test src/toil/test/src/realtimeLoggerTest.py::RealtimeLoggerTest::testRealtimeLogger [gw3] [ 98%] PASSED src/toil/test/src/threadingTest.py::ThreadingTest::testGlobalMutexOrdering src/toil/test/src/threadingTest.py::ThreadingTest::testLastProcessStanding [gw2] [ 98%] FAILED src/toil/test/src/helloWorldTest.py::HelloWorldTest::testHelloWorld src/toil/test/src/promisesTest.py::CachedUnpicklingJobStoreTest::test [gw2] [ 98%] FAILED src/toil/test/src/promisesTest.py::CachedUnpicklingJobStoreTest::test src/toil/test/src/promisesTest.py::ChainedIndexedPromisesTest::test [gw1] [ 98%] PASSED src/toil/test/src/realtimeLoggerTest.py::RealtimeLoggerTest::testRealtimeLogger src/toil/test/src/workerTest.py::WorkerTests::testNextChainable [gw1] [ 98%] PASSED src/toil/test/src/workerTest.py::WorkerTests::testNextChainable [gw0] [ 99%] PASSED src/toil/test/utils/toilDebugTest.py::DebugJobTest::test_run_job src/toil/test/jobStores/jobStoreTest.py::InvalidAWSJobStoreTest::testInvalidJobStoreName [gw0] [ 99%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::InvalidAWSJobStoreTest::testInvalidJobStoreName src/toil/test/src/systemTest.py::SystemTest::testAtomicityOfNonEmptyDirectoryRenames [gw2] [ 99%] FAILED src/toil/test/src/promisesTest.py::ChainedIndexedPromisesTest::test [gw0] [ 99%] PASSED src/toil/test/src/systemTest.py::SystemTest::testAtomicityOfNonEmptyDirectoryRenames [gw3] [ 99%] PASSED src/toil/test/src/threadingTest.py::ThreadingTest::testLastProcessStanding src/toil/test/batchSystems/test_lsf_helper.py::LSFHelperTest::test_parse_mem_and_cmd_from_output [gw3] [100%] PASSED src/toil/test/batchSystems/test_lsf_helper.py::LSFHelperTest::test_parse_mem_and_cmd_from_output =================================== FAILURES =================================== ___________ CachingFileStoreTestWithFileJobStore.testDeleteLocalFile ___________ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 1229, in testDeleteLocalFile Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_deleteLocalFileFn' kind-_deleteLocalFileFn/instance-o9q7bz7p v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testDeleteLocalFile (Apr 30 2024 11:56:54:518240 PDT) ------------------------------ Captured log call ------------------------------- DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testDeleteLocalFile-p_35157n/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testDeleteLocalFile-p_35157n/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-qzkz3m87', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testDeleteLocalFile-p_35157n', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-qzkz3m87'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '20f74281-8825-4979-9687-c2d90f7d63ea' DEBUG toil.common:common.py:740 Obtained node ID 9d1f8bcc-f705-4f67-84fb-c68d8c6c626b from file /proc/sys/kernel/random/boot_id DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testDeleteLocalFile-p_35157n/toilwf-f5e9394877e15c2988f327ce9841c87b DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3024641264. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3024641264 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-1550670320de43ba8cce2ced2621baa2/stream for result of '_deleteLocalFileFn' TemporaryID(c1f6101a-9a77-405f-8d2a-e3a04e377c90) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_deleteLocalFileFn' kind-_deleteLocalFileFn/instance-o9q7bz7p v0 DEBUG toil.job:job.py:1299 New job version: '_deleteLocalFileFn' kind-_deleteLocalFileFn/instance-o9q7bz7p v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '20f74281-8825-4979-9687-c2d90f7d63ea', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-qzkz3m87', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testDeleteLocalFile-p_35157n', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_deleteLocalFileFn/instance-o9q7bz7p, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_deleteLocalFileFn/instance-o9q7bz7p', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_deleteLocalFileFn' kind-_deleteLocalFileFn/instance-o9q7bz7p v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9970765113830566 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.0033864974975585938 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3024641264... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3024641264 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-f5e9394877e15c2988f327ce9841c87b/deferred/func43nhkdlv DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-f5e9394877e15c2988f327ce9841c87b/deferred/func43nhkdlv DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testDeleteLocalFile-p_35157n/toilwf-f5e9394877e15c2988f327ce9841c87b DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-f5e9394877e15c2988f327ce9841c87b DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.08661937713623047 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-qzkz3m87) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testDeleteLocalFile _____ CachingFileStoreTestWithFileJobStore.testReadCachHitFileFromJobStore _____ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 805, in testReadCachHitFileFromJobStore Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-s15muwgp v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testReadCachHitFileFromJobStore (Apr 30 2024 11:56:58:340916 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadCachHitFileFromJobStore ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCachHitFileFromJobStore-qnm6_dyb/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCachHitFileFromJobStore-qnm6_dyb/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-5y36l1xx', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCachHitFileFromJobStore-qnm6_dyb', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-5y36l1xx'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'c63225b5-cb92-462d-8a23-948cf8da67c7' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCachHitFileFromJobStore-qnm6_dyb/toilwf-631e6f85b3415391ac0ae635fb7c15a5 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3025268904. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3025268904 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-4ebc4ab181024f04b27ec41c5905d851/stream for result of '_writeFileToJobStoreWithAsserts' TemporaryID(3f8d1010-402f-44dd-968a-5db09ed56b0f) v0 DEBUG toil.job:job.py:2624 Saving graph of 2 jobs, 2 non-service, 2 new DEBUG toil.job:job.py:2644 Processing job '_readFromJobStore' kind-_readFromJobStore/instance-rz34665k v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-12850960a3dd4e409f37c1d61ed94585/stream for result of '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-s15muwgp v0 DEBUG toil.job:job.py:2644 Processing job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-s15muwgp v0 DEBUG toil.job:job.py:1299 New job version: '_readFromJobStore' kind-_readFromJobStore/instance-rz34665k v1 DEBUG toil.job:job.py:1299 New job version: '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-s15muwgp v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'c63225b5-cb92-462d-8a23-948cf8da67c7', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-5y36l1xx', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCachHitFileFromJobStore-qnm6_dyb', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_writeFileToJobStoreWithAsserts/instance-s15muwgp, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_writeFileToJobStoreWithAsserts/instance-s15muwgp', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-s15muwgp v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9949018955230713 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.0022792816162109375 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3025268904... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3025268904 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-631e6f85b3415391ac0ae635fb7c15a5/deferred/funcj4pl3wsh DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-631e6f85b3415391ac0ae635fb7c15a5/deferred/funcj4pl3wsh DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCachHitFileFromJobStore-qnm6_dyb/toilwf-631e6f85b3415391ac0ae635fb7c15a5 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-631e6f85b3415391ac0ae635fb7c15a5 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.01634836196899414 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-5y36l1xx) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadCachHitFileFromJobStore _ CachingFileStoreTestWithFileJobStore.testReadCacheMissFileFromJobStoreWithCachingReadFile _ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 735, in testReadCacheMissFileFromJobStoreWithCachingReadFile self._testCacheMissFunction(cacheReadFile=True) File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 749, in _testCacheMissFunction Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-0f1uzl5j v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testReadCacheMissFileFromJobStoreWithCachingReadFile (Apr 30 2024 11:57:01:267812 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadCacheMissFileFromJobStoreWithCachingReadFile ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithCachingReadFile-kqn0hdo2/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithCachingReadFile-kqn0hdo2/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-nf1_y80e', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithCachingReadFile-kqn0hdo2', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-nf1_y80e'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'd4cb318a-e8f7-4241-8d95-06512b58830c' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithCachingReadFile-kqn0hdo2/toilwf-2f3a64084dc357e8957bb155953e2e3a DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3005593904. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3005593904 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-ec5e4e67854043158284159b895de964/stream for result of '_writeFileToJobStoreWithAsserts' TemporaryID(a8203cb3-b16f-473c-8920-eb7c4c9eee34) v0 DEBUG toil.job:job.py:2624 Saving graph of 2 jobs, 2 non-service, 2 new DEBUG toil.job:job.py:2644 Processing job '_readFromJobStore' kind-_readFromJobStore/instance-gh9d3tff v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-a34ca54b9faf49ea962cec6c9f53785d/stream for result of '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-0f1uzl5j v0 DEBUG toil.job:job.py:2644 Processing job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-0f1uzl5j v0 DEBUG toil.job:job.py:1299 New job version: '_readFromJobStore' kind-_readFromJobStore/instance-gh9d3tff v1 DEBUG toil.job:job.py:1299 New job version: '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-0f1uzl5j v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'd4cb318a-e8f7-4241-8d95-06512b58830c', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-nf1_y80e', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithCachingReadFile-kqn0hdo2', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_writeFileToJobStoreWithAsserts/instance-0f1uzl5j, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_writeFileToJobStoreWithAsserts/instance-0f1uzl5j', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-0f1uzl5j v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9967186450958252 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5022461414337158 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3005593904... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3005593904 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-2f3a64084dc357e8957bb155953e2e3a/deferred/funcn5oncta3 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-2f3a64084dc357e8957bb155953e2e3a/deferred/funcn5oncta3 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithCachingReadFile-kqn0hdo2/toilwf-2f3a64084dc357e8957bb155953e2e3a DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-2f3a64084dc357e8957bb155953e2e3a DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.018924474716186523 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-nf1_y80e) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadCacheMissFileFromJobStoreWithCachingReadFile _ CachingFileStoreTestWithFileJobStore.testReadCacheMissFileFromJobStoreWithoutCachingReadFile _ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 728, in testReadCacheMissFileFromJobStoreWithoutCachingReadFile self._testCacheMissFunction(cacheReadFile=False) File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 749, in _testCacheMissFunction Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-f14r16la v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testReadCacheMissFileFromJobStoreWithoutCachingReadFile (Apr 30 2024 11:57:04:646851 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadCacheMissFileFromJobStoreWithoutCachingReadFile ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithoutCachingReadFile-dlgatw2q/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithoutCachingReadFile-dlgatw2q/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-s41lldp8', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithoutCachingReadFile-dlgatw2q', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-s41lldp8'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'e6b9e922-e351-48cd-a4d6-b9564a4853f0' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithoutCachingReadFile-dlgatw2q/toilwf-cd7c645ac1fa59f09abff1f5ec8630d7 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 2995612648. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 2995612648 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-eeba6b65f7734270babe9ed5362634ce/stream for result of '_writeFileToJobStoreWithAsserts' TemporaryID(0522b4d1-0b4e-46bd-85eb-b6185d22225b) v0 DEBUG toil.job:job.py:2624 Saving graph of 2 jobs, 2 non-service, 2 new DEBUG toil.job:job.py:2644 Processing job '_readFromJobStore' kind-_readFromJobStore/instance-_frbetxw v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-c6c04a2fde754c6f9cae3b6273a076fb/stream for result of '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-f14r16la v0 DEBUG toil.job:job.py:2644 Processing job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-f14r16la v0 DEBUG toil.job:job.py:1299 New job version: '_readFromJobStore' kind-_readFromJobStore/instance-_frbetxw v1 DEBUG toil.job:job.py:1299 New job version: '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-f14r16la v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'e6b9e922-e351-48cd-a4d6-b9564a4853f0', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-s41lldp8', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithoutCachingReadFile-dlgatw2q', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_writeFileToJobStoreWithAsserts/instance-f14r16la, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_writeFileToJobStoreWithAsserts/instance-f14r16la', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-f14r16la v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9962289333343506 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5021073818206787 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 2995612648... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 2995612648 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-cd7c645ac1fa59f09abff1f5ec8630d7/deferred/funcz5nmk3pr DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-cd7c645ac1fa59f09abff1f5ec8630d7/deferred/funcz5nmk3pr DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithoutCachingReadFile-dlgatw2q/toilwf-cd7c645ac1fa59f09abff1f5ec8630d7 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-cd7c645ac1fa59f09abff1f5ec8630d7 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.025791168212890625 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-s41lldp8) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadCacheMissFileFromJobStoreWithoutCachingReadFile __________________________ JobTest.testDAGConsistency __________________________ [gw0] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobTest.py", line 145, in testDAGConsistency toil.start(i) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'parent' kind-parent/instance-xirriuyo v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.jobTest.JobTest:testDAGConsistency (Apr 30 2024 11:57:07:112512 PDT) ------------------------------ Captured log call ------------------------------- DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'charset_normalizer', 'docker', 'urllib3', 'requests', 'botocore', 'asyncio', 'boto3', 'boto', 'bcdocs', 'debugWorkflow', 'websocket', 'concurrent', 'dill'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'debug', 'logFile': None, 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.jobTest.JobTest-testDAGConsistency-owml5gxi/jobStore', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.jobTest.JobTest-testDAGConsistency-owml5gxi/jobStore'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'e30dfe1d-168a-46a6-8eff-6eab60fe9f13' DEBUG toil.common:common.py:740 Obtained node ID 9d1f8bcc-f705-4f67-84fb-c68d8c6c626b from file /proc/sys/kernel/random/boot_id DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toilwf-bb940987b7bf5a4ea42831d8d8a51627 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3023674824. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3023674824 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.jobTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-547f7a3b4f054ebebdf0ea6e438f94ce/stream for result of 'parent' TemporaryID(01070cbb-e1dc-4bee-90a1-c79c9c050599) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job 'parent' kind-parent/instance-xirriuyo v0 DEBUG toil.job:job.py:1299 New job version: 'parent' kind-parent/instance-xirriuyo v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'e30dfe1d-168a-46a6-8eff-6eab60fe9f13', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.jobTest.JobTest-testDAGConsistency-owml5gxi/jobStore', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'debug', 'workflowAttemptNumber': 0} DEBUG toil.realtimeLogger:realtimeLogger.py:162 Real-time logging disabled DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-parent/instance-xirriuyo, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-parent/instance-xirriuyo', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job 'parent' kind-parent/instance-xirriuyo v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.99802565574646 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.0038251876831054688 seconds DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3023674824... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3023674824 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-bb940987b7bf5a4ea42831d8d8a51627/deferred/funcakl3dj2q DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-bb940987b7bf5a4ea42831d8d8a51627/deferred/funcakl3dj2q DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toilwf-bb940987b7bf5a4ea42831d8d8a51627 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-bb940987b7bf5a4ea42831d8d8a51627 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.02158641815185547 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.jobTest.JobTest-testDAGConsistency-owml5gxi/jobStore) INFO toil.test:__init__.py:120 Tore down toil.test.src.jobTest.JobTest.testDAGConsistency _____ CachingFileStoreTestWithFileJobStore.testReadWriteFileStreamTextMode _____ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 351, in testReadWriteFileStreamTextMode toil.start(A) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-bj510p94 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testReadWriteFileStreamTextMode (Apr 30 2024 11:57:08:049905 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadWriteFileStreamTextMode ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-v9x11n1w/logFile'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-v9x11n1w/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-23g4tpoz', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-v9x11n1w', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-23g4tpoz'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'e0d7985a-0e83-4a22-b74f-fe1901e140bc' DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-v9x11n1w/toilwf-7697722b6d1c5e80b743d40f263ab9cd DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 2995358312. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 2995358312 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-85748ff0376543a5b9cf62a3f92494c1/stream for result of '_testReadWriteFileStreamTextMode' TemporaryID(412e60e4-4682-406d-a224-e09b94ceaeac) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-bj510p94 v0 DEBUG toil.job:job.py:1299 New job version: '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-bj510p94 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'e0d7985a-0e83-4a22-b74f-fe1901e140bc', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-23g4tpoz', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-v9x11n1w', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_testReadWriteFileStreamTextMode/instance-bj510p94, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_testReadWriteFileStreamTextMode/instance-bj510p94', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-bj510p94 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 2.0017964839935303 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.49666929244995117 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 2995358312... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 2995358312 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-7697722b6d1c5e80b743d40f263ab9cd/deferred/funcjlv3xitw DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-7697722b6d1c5e80b743d40f263ab9cd/deferred/funcjlv3xitw DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-v9x11n1w/toilwf-7697722b6d1c5e80b743d40f263ab9cd DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-7697722b6d1c5e80b743d40f263ab9cd DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.023721933364868164 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-23g4tpoz) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadWriteFileStreamTextMode ___ CachingFileStoreTestWithFileJobStore.testSimultaneousReadsUncachedStream ___ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 1302, in testSimultaneousReadsUncachedStream Job.Runner.startToil(parent, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testSimultaneousReadsUncachedStream (Apr 30 2024 11:57:11:473836 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testSimultaneousReadsUncachedStream ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testSimultaneousReadsUncachedStream-4e_ta5wu/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testSimultaneousReadsUncachedStream-4e_ta5wu/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-kk5k5d1t', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testSimultaneousReadsUncachedStream-4e_ta5wu', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': True, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-kk5k5d1t'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '018efef1-4582-4e04-96ab-99a5f1900eff' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testSimultaneousReadsUncachedStream-4e_ta5wu/toilwf-1890dead38bc5804a2ead3586ae426a2 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 2995537096. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 2995537096 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-2ddfa17cf9874843a77f337ddd1b2114/stream for result of '_createUncachedFileStream' TemporaryID(b86bbc9f-02c9-4644-90e5-a9ab7fbf19c6) v0 DEBUG toil.job:job.py:2624 Saving graph of 31 jobs, 31 non-service, 31 new DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-xy_rsc89 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-9e0c6043bd8149d0bb5148b758ff5aee/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-6pfsjsq7 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-9e21ecc7a1864863a41cf22ebda947ea/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-9risljqk v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-96fde190a502461191ab381b4e8d514a/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-eeh7oqiu v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-23e1076115ff4f6f867ed3ce6a13a870/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-40891fjj v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-3838d30c4c1f4af88ed7a29ea8ef4337/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-aw_c4_du v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-090019dde64d415daddf6bd64fcaba4f/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-omeewans v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-09521983c83e409ab5d3aca365fb7ff3/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-mbg261kn v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-b854d5f1bbb845feaf850204aac29838/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-4jhxojz4 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-9023b94e70564e34af12b32526d70ffc/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-wg6_sd9r v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-ea8363b033df4c5fae9f27d549a2b608/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-e8d25dhf v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-e9fd3163f0944fb0af0caa9c1240a36a/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-stwji5l2 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-af45d1171f804a749896114ecb1e6c7f/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-enh04ms3 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-7abe4258db2f4bbfb204ec119652f8f1/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-bhpb9h62 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-9b868b46c2cf4cefb64b51e70f0b91c3/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-4k4s07px v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-4bd19606d8634bc09daec1ac63a4628f/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-2fd2ck7g v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-75807ccfa6394163b1e112a0d25e02a0/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-zsbs6x73 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-d8d9de0c3a3b4ffcb586d9de4c00ee2c/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-tndt08vt v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-d5464e70848f4c958c3f58b5872c7591/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-61iek9tv v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-32eb22eec09b474786a7d34dcfbf1ee3/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-rlwge4vz v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-9f9472a1a05d44a8841f11a981404966/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-d1wttaat v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-90c836b2f71447d983c749032c46fc0a/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-dqt5u9ax v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-6b3068958f7a4c12aa8079e8a65e2d73/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-b2w0dme7 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-18c36a80b5084a74ad74bb8cfb4f8aa3/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-_fs_vahy v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-c30d33cef8bb4aca9baeebc0f067191b/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-wx0rh7oe v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-1bd0c6b52ef743e7b8ab1457095cc722/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-nny8d_j3 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-f54161985fdd46b8aa4a223da92b7f7c/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-mg8t_249 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-226e03646518450aaad0f8a3b37657f2/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-wdulab5g v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-82981a36cce84568adb568d3553614e7/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-jjn9tgod v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-7ef852d449164554ba2313281b34ff3c/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-h87pe3rs v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-b07850b9b84f417eb38fd82a6c9ada16/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:2644 Processing job '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v0 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-xy_rsc89 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-6pfsjsq7 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-9risljqk v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-eeh7oqiu v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-40891fjj v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-aw_c4_du v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-omeewans v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-mbg261kn v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-4jhxojz4 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-wg6_sd9r v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-e8d25dhf v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-stwji5l2 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-enh04ms3 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-bhpb9h62 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-4k4s07px v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-2fd2ck7g v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-zsbs6x73 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-tndt08vt v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-61iek9tv v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-rlwge4vz v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-d1wttaat v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-dqt5u9ax v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-b2w0dme7 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-_fs_vahy v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-wx0rh7oe v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-nny8d_j3 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-mg8t_249 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-wdulab5g v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-jjn9tgod v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-h87pe3rs v1 DEBUG toil.job:job.py:1299 New job version: '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '018efef1-4582-4e04-96ab-99a5f1900eff', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-kk5k5d1t', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testSimultaneousReadsUncachedStream-4e_ta5wu', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': True, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_createUncachedFileStream/instance-sphhjx39, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_createUncachedFileStream/instance-sphhjx39', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_createUncachedFileStream' kind-_createUncachedFileStream/instance-sphhjx39 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9956152439117432 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5032968521118164 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 2995537096... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 2995537096 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-1890dead38bc5804a2ead3586ae426a2/deferred/funce_i9y661 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-1890dead38bc5804a2ead3586ae426a2/deferred/funce_i9y661 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testSimultaneousReadsUncachedStream-4e_ta5wu/toilwf-1890dead38bc5804a2ead3586ae426a2 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-1890dead38bc5804a2ead3586ae426a2 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.04184532165527344 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-kk5k5d1t) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testSimultaneousReadsUncachedStream ___________ CachingFileStoreTestWithFileJobStore.testToilIsNotBroken ___________ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 120, in testToilIsNotBroken Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_uselessFunc' kind-_uselessFunc/instance-cvy8b0lc v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testToilIsNotBroken (Apr 30 2024 11:57:15:427055 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testToilIsNotBroken ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testToilIsNotBroken-6nj6srlq/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testToilIsNotBroken-6nj6srlq/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-2yizy0n1', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testToilIsNotBroken-6nj6srlq', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-2yizy0n1'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'f5f2e344-d26f-4e18-ac4b-9e88f4cab0e6' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testToilIsNotBroken-6nj6srlq/toilwf-787173c6156f515291b7d6859cb0a2dd DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3005535888. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3005535888 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-eee15e8313a64e55a4d28ffda9b19a4f/stream for result of '_uselessFunc' TemporaryID(d182a0c4-d739-4cdc-bddf-39e803c5fe43) v0 DEBUG toil.job:job.py:2624 Saving graph of 4 jobs, 4 non-service, 4 new DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-k291xdkn v0 DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-_uf_1utp v0 DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-i7waw3sk v0 DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-cvy8b0lc v0 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-k291xdkn v1 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-_uf_1utp v1 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-i7waw3sk v1 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-cvy8b0lc v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'f5f2e344-d26f-4e18-ac4b-9e88f4cab0e6', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-2yizy0n1', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testToilIsNotBroken-6nj6srlq', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_uselessFunc/instance-cvy8b0lc, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_uselessFunc/instance-cvy8b0lc', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_uselessFunc' kind-_uselessFunc/instance-cvy8b0lc v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9948434829711914 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5019991397857666 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3005535888... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3005535888 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-787173c6156f515291b7d6859cb0a2dd/deferred/func7j736tcw DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-787173c6156f515291b7d6859cb0a2dd/deferred/func7j736tcw DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testToilIsNotBroken-6nj6srlq/toilwf-787173c6156f515291b7d6859cb0a2dd DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-787173c6156f515291b7d6859cb0a2dd DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.02935957908630371 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-2yizy0n1) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testToilIsNotBroken _____________________________ JobTest.testStatic2 ______________________________ [gw0] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobTest.py", line 115, in testStatic2 Job.Runner.startToil(A, options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'fn1Test' kind-fn1Test/instance-0xahv_d7 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.jobTest.JobTest:testStatic2 (Apr 30 2024 11:57:17:234315 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.jobTest.JobTest.testStatic2 ... DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.jobTest.JobTest.testStatic2 ____ CachingFileStoreTestWithFileJobStore.testWriteExportFileCompatibility _____ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 300, in testWriteExportFileCompatibility initialPermissions, fileID = toil.start(export_file_job) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-tirpsn85 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testWriteExportFileCompatibility (Apr 30 2024 11:57:18:882656 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteExportFileCompatibility ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-d4c2iev5/logFile'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-d4c2iev5/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-gduo1f8e', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-d4c2iev5', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-gduo1f8e'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'a15caad7-d28b-4fed-8c10-37aefe3616b1' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-d4c2iev5/toilwf-8472eeb3e3f651e183faac3f7d3c485a DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3024682896. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3024682896 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-e7543b48f8a24208a45a895bf6e09dac/stream for result of '_testWriteExportFileCompatibility' TemporaryID(ba343a75-f3ad-4b08-a16f-4f774196bbe5) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-tirpsn85 v0 DEBUG toil.job:job.py:1299 New job version: '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-tirpsn85 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'a15caad7-d28b-4fed-8c10-37aefe3616b1', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-gduo1f8e', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-d4c2iev5', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_testWriteExportFileCompatibility/instance-tirpsn85, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_testWriteExportFileCompatibility/instance-tirpsn85', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-tirpsn85 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.994567632675171 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5041835308074951 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3024682896... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3024682896 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-8472eeb3e3f651e183faac3f7d3c485a/deferred/func7cdms4y9 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-8472eeb3e3f651e183faac3f7d3c485a/deferred/func7cdms4y9 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-d4c2iev5/toilwf-8472eeb3e3f651e183faac3f7d3c485a DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-8472eeb3e3f651e183faac3f7d3c485a DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.05371832847595215 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-gduo1f8e) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteExportFileCompatibility ___ NonCachingFileStoreTestWithFileJobStore.testReadWriteFileStreamTextMode ____ [gw0] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 351, in testReadWriteFileStreamTextMode toil.start(A) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-f0l3teev v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore:testReadWriteFileStreamTextMode (Apr 30 2024 11:57:19:828999 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testReadWriteFileStreamTextMode ... DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'charset_normalizer', 'docker', 'urllib3', 'requests', 'botocore', 'asyncio', 'boto3', 'boto', 'bcdocs', 'debugWorkflow', 'websocket', 'concurrent', 'dill'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-rvpf184s/logFile'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-rvpf184s/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-883cfhup', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-rvpf184s', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-883cfhup'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '4dcec815-19f3-46d2-bada-4adb64e242d3' DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-rvpf184s/toilwf-46dc704e93f85da19645bad5f4e744d0 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3028685976. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3028685976 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-2dd4ee9ce84740bfbb0070fc81f94413/stream for result of '_testReadWriteFileStreamTextMode' TemporaryID(91c4efc8-ceea-461b-9550-11aa4b5bf874) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-f0l3teev v0 DEBUG toil.job:job.py:1299 New job version: '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-f0l3teev v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '4dcec815-19f3-46d2-bada-4adb64e242d3', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-883cfhup', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-rvpf184s', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_testReadWriteFileStreamTextMode/instance-f0l3teev, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_testReadWriteFileStreamTextMode/instance-f0l3teev', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-f0l3teev v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9980015754699707 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.0019407272338867188 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3028685976... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3028685976 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-46dc704e93f85da19645bad5f4e744d0/deferred/funcqoiu4_so DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-46dc704e93f85da19645bad5f4e744d0/deferred/funcqoiu4_so DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-rvpf184s/toilwf-46dc704e93f85da19645bad5f4e744d0 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-46dc704e93f85da19645bad5f4e744d0 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.021320104598999023 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-883cfhup) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testReadWriteFileStreamTextMode _________ NonCachingFileStoreTestWithFileJobStore.testToilIsNotBroken __________ [gw0] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 120, in testToilIsNotBroken Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_uselessFunc' kind-_uselessFunc/instance-0vseyvob v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore:testToilIsNotBroken (Apr 30 2024 11:57:22:666452 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testToilIsNotBroken ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'charset_normalizer', 'docker', 'urllib3', 'requests', 'botocore', 'asyncio', 'boto3', 'boto', 'bcdocs', 'debugWorkflow', 'websocket', 'concurrent', 'dill'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testToilIsNotBroken-c1_9ksch/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'charset_normalizer', 'docker', 'urllib3', 'requests', 'botocore', 'asyncio', 'boto3', 'boto', 'bcdocs', 'debugWorkflow', 'websocket', 'concurrent', 'dill'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testToilIsNotBroken-c1_9ksch/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-h6xzrnkb', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testToilIsNotBroken-c1_9ksch', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-h6xzrnkb'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '1789c82a-7398-444e-a482-9eb9f0612af6' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testToilIsNotBroken-c1_9ksch/toilwf-59d29270ce365b229909032612e6fdf6 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3023828840. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3023828840 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-baafc785cef74cf5985a845d0d1e94bc/stream for result of '_uselessFunc' TemporaryID(2d028801-a9b6-4653-b53b-33ff265bb121) v0 DEBUG toil.job:job.py:2624 Saving graph of 4 jobs, 4 non-service, 4 new DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-n2ex0hnu v0 DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-q40rnxvy v0 DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-hmpitsh2 v0 DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-0vseyvob v0 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-n2ex0hnu v1 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-q40rnxvy v1 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-hmpitsh2 v1 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-0vseyvob v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '1789c82a-7398-444e-a482-9eb9f0612af6', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-h6xzrnkb', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testToilIsNotBroken-c1_9ksch', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_uselessFunc/instance-0vseyvob, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_uselessFunc/instance-0vseyvob', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_uselessFunc' kind-_uselessFunc/instance-0vseyvob v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9974992275238037 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.007719993591308594 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3023828840... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3023828840 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-59d29270ce365b229909032612e6fdf6/deferred/func6rk125zg DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-59d29270ce365b229909032612e6fdf6/deferred/func6rk125zg DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testToilIsNotBroken-c1_9ksch/toilwf-59d29270ce365b229909032612e6fdf6 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-59d29270ce365b229909032612e6fdf6 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.030945301055908203 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-h6xzrnkb) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testToilIsNotBroken ______ CachingFileStoreTestWithFileJobStore.testWriteLocalFileToJobStore _______ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 719, in testWriteLocalFileToJobStore Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-rr61mv1i v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testWriteLocalFileToJobStore (Apr 30 2024 11:57:22:309894 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteLocalFileToJobStore ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteLocalFileToJobStore-aeb2tfu0/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteLocalFileToJobStore-aeb2tfu0/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-ifjo7255', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteLocalFileToJobStore-aeb2tfu0', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-ifjo7255'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '667bef1e-7490-4727-b65e-c5f82c13b01d' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteLocalFileToJobStore-aeb2tfu0/toilwf-5347ed42336f59fdb797a75a1da25011 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3024583632. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3024583632 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-c92891ead8f24ec7bae2e1f8e9c1200b/stream for result of '_writeFileToJobStoreWithAsserts' TemporaryID(44550f86-ec98-45b2-a769-bef22d976743) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-rr61mv1i v0 DEBUG toil.job:job.py:1299 New job version: '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-rr61mv1i v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '667bef1e-7490-4727-b65e-c5f82c13b01d', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-ifjo7255', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteLocalFileToJobStore-aeb2tfu0', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_writeFileToJobStoreWithAsserts/instance-rr61mv1i, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_writeFileToJobStoreWithAsserts/instance-rr61mv1i', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-rr61mv1i v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9941844940185547 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5041041374206543 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3024583632... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3024583632 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-5347ed42336f59fdb797a75a1da25011/deferred/func7mhtx4rp DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-5347ed42336f59fdb797a75a1da25011/deferred/func7mhtx4rp DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteLocalFileToJobStore-aeb2tfu0/toilwf-5347ed42336f59fdb797a75a1da25011 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-5347ed42336f59fdb797a75a1da25011 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.027444124221801758 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-ifjo7255) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteLocalFileToJobStore ___ NonCachingFileStoreTestWithFileJobStore.testWriteExportFileCompatibility ___ [gw0] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 300, in testWriteExportFileCompatibility initialPermissions, fileID = toil.start(export_file_job) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-l0n9ul3_ v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore:testWriteExportFileCompatibility (Apr 30 2024 11:57:25:546168 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testWriteExportFileCompatibility ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'charset_normalizer', 'docker', 'urllib3', 'requests', 'botocore', 'asyncio', 'boto3', 'boto', 'bcdocs', 'debugWorkflow', 'websocket', 'concurrent', 'dill'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-ow3uj26x/logFile'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-ow3uj26x/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-r2g4d6gc', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-ow3uj26x', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-r2g4d6gc'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '564d99e3-645f-4d1b-a3c6-db38b7a8db02' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-ow3uj26x/toilwf-9354d03f23c057d69ac493253e431c5f DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3024331592. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3024331592 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-a6a379f64dad4a2bbc0c1eec899577a4/stream for result of '_testWriteExportFileCompatibility' TemporaryID(c8acd29d-3235-412d-8fab-17f652d0fc57) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-l0n9ul3_ v0 DEBUG toil.job:job.py:1299 New job version: '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-l0n9ul3_ v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '564d99e3-645f-4d1b-a3c6-db38b7a8db02', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-r2g4d6gc', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-ow3uj26x', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_testWriteExportFileCompatibility/instance-l0n9ul3_, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_testWriteExportFileCompatibility/instance-l0n9ul3_', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-l0n9ul3_ v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9974675178527832 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.0021500587463378906 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3024331592... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3024331592 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-9354d03f23c057d69ac493253e431c5f/deferred/func4f_qx76p DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-9354d03f23c057d69ac493253e431c5f/deferred/func4f_qx76p DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-ow3uj26x/toilwf-9354d03f23c057d69ac493253e431c5f DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-9354d03f23c057d69ac493253e431c5f DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.044164419174194336 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-r2g4d6gc) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testWriteExportFileCompatibility _____ CachingFileStoreTestWithFileJobStore.testWriteNonLocalFileToJobStore _____ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 711, in testWriteNonLocalFileToJobStore Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-cy8k5gn8 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testWriteNonLocalFileToJobStore (Apr 30 2024 11:57:25:709714 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteNonLocalFileToJobStore ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteNonLocalFileToJobStore-70dqo5kg/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteNonLocalFileToJobStore-70dqo5kg/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-iwnucrqn', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteNonLocalFileToJobStore-70dqo5kg', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-iwnucrqn'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'ded03681-1e56-47cd-a7b7-f3d5690c38ef' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteNonLocalFileToJobStore-70dqo5kg/toilwf-39130cae91db50f68070579ec3d3ca77 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3005479168. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3005479168 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-9d3a1ba4e676452da26dafe3205a8531/stream for result of '_writeFileToJobStoreWithAsserts' TemporaryID(bd5e0dbf-d6ff-470a-b1be-de50da7f6c59) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-cy8k5gn8 v0 DEBUG toil.job:job.py:1299 New job version: '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-cy8k5gn8 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'ded03681-1e56-47cd-a7b7-f3d5690c38ef', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-iwnucrqn', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteNonLocalFileToJobStore-70dqo5kg', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_writeFileToJobStoreWithAsserts/instance-cy8k5gn8, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_writeFileToJobStoreWithAsserts/instance-cy8k5gn8', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-cy8k5gn8 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9952054023742676 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5031793117523193 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3005479168... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3005479168 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-39130cae91db50f68070579ec3d3ca77/deferred/funcgf5wrgsf DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-39130cae91db50f68070579ec3d3ca77/deferred/funcgf5wrgsf DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteNonLocalFileToJobStore-70dqo5kg/toilwf-39130cae91db50f68070579ec3d3ca77 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-39130cae91db50f68070579ec3d3ca77 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.05561637878417969 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-iwnucrqn) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteNonLocalFileToJobStore __ NonCachingFileStoreTestWithFileJobStore.testWriteReadGlobalFilePermissions __ [gw0] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 269, in testWriteReadGlobalFilePermissions Job.Runner.startToil(read_write_job, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-su8dfm9a v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore:testWriteReadGlobalFilePermissions (Apr 30 2024 11:57:28:404773 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testWriteReadGlobalFilePermissions ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'charset_normalizer', 'docker', 'urllib3', 'requests', 'botocore', 'asyncio', 'boto3', 'boto', 'bcdocs', 'debugWorkflow', 'websocket', 'concurrent', 'dill'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-rrrh78mk/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'charset_normalizer', 'docker', 'urllib3', 'requests', 'botocore', 'asyncio', 'boto3', 'boto', 'bcdocs', 'debugWorkflow', 'websocket', 'concurrent', 'dill'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-rrrh78mk/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-g_frrs3u', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-rrrh78mk', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-g_frrs3u'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '15dbf88c-2467-4e8d-9c82-1dd3c9d99608' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-rrrh78mk/toilwf-7dc12dd9d78159488b03641f7fcec7e7 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3024204736. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3024204736 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-11c63d8e719d4009912e41aa200a67c7/stream for result of '_testWriteReadGlobalFilePermissions' TemporaryID(0f7a278f-a823-4249-904f-9124014fcb49) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-su8dfm9a v0 DEBUG toil.job:job.py:1299 New job version: '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-su8dfm9a v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '15dbf88c-2467-4e8d-9c82-1dd3c9d99608', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-g_frrs3u', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-rrrh78mk', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_testWriteReadGlobalFilePermissions/instance-su8dfm9a, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_testWriteReadGlobalFilePermissions/instance-su8dfm9a', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-su8dfm9a v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9971792697906494 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.0025718212127685547 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3024204736... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3024204736 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-7dc12dd9d78159488b03641f7fcec7e7/deferred/funcei4ux3db DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-7dc12dd9d78159488b03641f7fcec7e7/deferred/funcei4ux3db DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-rrrh78mk/toilwf-7dc12dd9d78159488b03641f7fcec7e7 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-7dc12dd9d78159488b03641f7fcec7e7 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.037030935287475586 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-g_frrs3u) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testWriteReadGlobalFilePermissions ___ CachingFileStoreTestWithFileJobStore.testWriteReadGlobalFilePermissions ____ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 269, in testWriteReadGlobalFilePermissions Job.Runner.startToil(read_write_job, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-b8ut2aoa v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testWriteReadGlobalFilePermissions (Apr 30 2024 11:57:29:132152 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteReadGlobalFilePermissions ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-7k1vs8_0/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-7k1vs8_0/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-0g70k6di', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-7k1vs8_0', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-0g70k6di'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '70ef267d-3446-4d5a-b0db-f952dba495fb' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-7k1vs8_0/toilwf-3fc2381b0a5b5601b1097b94e1767174 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 2995537984. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 2995537984 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-9135a89bbd624b53a6cca4fa753db031/stream for result of '_testWriteReadGlobalFilePermissions' TemporaryID(901bb32c-7f10-4b3b-8840-8da0b622e0c0) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-b8ut2aoa v0 DEBUG toil.job:job.py:1299 New job version: '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-b8ut2aoa v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '70ef267d-3446-4d5a-b0db-f952dba495fb', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-0g70k6di', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-7k1vs8_0', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_testWriteReadGlobalFilePermissions/instance-b8ut2aoa, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_testWriteReadGlobalFilePermissions/instance-b8ut2aoa', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-b8ut2aoa v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9978513717651367 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.49817752838134766 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 2995537984... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 2995537984 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-3fc2381b0a5b5601b1097b94e1767174/deferred/func_6t0d_ak DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-3fc2381b0a5b5601b1097b94e1767174/deferred/func_6t0d_ak DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-7k1vs8_0/toilwf-3fc2381b0a5b5601b1097b94e1767174 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-3fc2381b0a5b5601b1097b94e1767174 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.033116817474365234 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-0g70k6di) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteReadGlobalFilePermissions ____________________ JobFileStoreTest.testCachingFileStore _____________________ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobFileStoreTest.py", line 38, in testCachingFileStore workflow.start(Job.wrapJobFn(simpleFileStoreJob)) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'simpleFileStoreJob' kind-simpleFileStoreJob/instance-byt4jmx9 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.jobFileStoreTest.JobFileStoreTest:testCachingFileStore (Apr 30 2024 11:57:38:775749 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.jobFileStoreTest.JobFileStoreTest.testCachingFileStore ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.jobFileStoreTest.JobFileStoreTest.testCachingFileStore _____________________ UtilsTest.testGetStatusFailedToilWF ______________________ [gw2] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/utilsTest.py", line 336, in testGetStatusFailedToilWF self.check_status('ERROR', status_fn=ToilStatus.getStatus) File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/utilsTest.py", line 312, in check_status self.assertEqual(s, status, f'Waited {seconds} seconds without status reaching {status}; stuck at {s}') File "/usr/lib/python3.12/unittest/case.py", line 885, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib/python3.12/unittest/case.py", line 1251, in assertMultiLineEqual self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib/python3.12/unittest/case.py", line 715, in fail raise self.failureException(msg) AssertionError: 'RUNNING' != 'ERROR' - RUNNING + ERROR : Waited 20 seconds without status reaching ERROR; stuck at RUNNING ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.utils.utilsTest.UtilsTest:testGetStatusFailedToilWF (Apr 30 2024 11:57:15:101498 PDT) ----------------------------- Captured stdout call ----------------------------- No sort file specified. Generating one automatically called: fileToSort.txt. ----------------------------- Captured stderr call ----------------------------- [2024-04-30T11:57:19-1200] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. [2024-04-30T11:57:19-1200] [MainThread] [I] [toil.leader] Issued job 'setup' kind-setup/instance-0y47mm1g v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 572.2 Mi, cores: 1, accelerators: [], preemptible: False [2024-04-30T11:57:21-1200] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run User defined signal 1 [2024-04-30T11:57:21-1200] [Thread-1 (daddy)] [E] [toil.batchSystems.singleMachine] Got exit code 138 (indicating failure) from job _toil_worker setup file:/tmp/toil-test-utils.utilsTest.UtilsTest-testGetStatusFailedToilWF-efmjuk5k/jobstore kind-setup/instance-0y47mm1g. [2024-04-30T11:57:21-1200] [MainThread] [W] [toil.leader] Job failed with exit value 138: 'setup' kind-setup/instance-0y47mm1g v1 Exit reason: None [2024-04-30T11:57:21-1200] [MainThread] [W] [toil.leader] No log file is present, despite job failing: 'setup' kind-setup/instance-0y47mm1g v1 [2024-04-30T11:57:21-1200] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'setup' kind-setup/instance-0y47mm1g v1 with ID kind-setup/instance-0y47mm1g to 1 [2024-04-30T11:57:21-1200] [MainThread] [W] [toil.job] We have increased the default memory of the failed job 'setup' kind-setup/instance-0y47mm1g v1 to 2147483648 bytes Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/build/reproducible-path/toil-6.1.0/src/toil/test/sort/sort.py", line 260, in main() File "/build/reproducible-path/toil-6.1.0/src/toil/test/sort/sort.py", line 248, in main sortedFileID = workflow.start(Job.wrapJobFn(setup, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'setup' kind-setup/instance-0y47mm1g v2 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. __________________________ UtilsTest.testPrintJobLog ___________________________ [gw2] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/usr/lib/python3.12/unittest/mock.py", line 1390, in patched return func(*newargs, **newkeywargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/utilsTest.py", line 376, in testPrintJobLog self.assertIn('invalidcommand', args[0]) File "/usr/lib/python3.12/unittest/case.py", line 1152, in assertIn self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib/python3.12/unittest/case.py", line 715, in fail raise self.failureException(msg) AssertionError: 'invalidcommand' not found in "LOG_FILE_OF_JOB: 'CWLJob' alwaysfails.cwl kind-CWLJob/instance-xlgawzkh v1 LOG: Job has no log file" ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.utils.utilsTest.UtilsTest:testPrintJobLog (Apr 30 2024 11:57:44:563301 PDT) ----------------------------- Captured stderr call ----------------------------- WARNING: You are using cwltool version 3.1.20240404144621, which is not the version Toil is tested against. To install the correct cwltool for Toil, do: pip install cwltool==3.1.20240112164112 [2024-04-30T11:57:49-1200] [MainThread] [I] [cwltool] Resolved 'src/toil/test/cwl/alwaysfails.cwl' to 'file:///build/reproducible-path/toil-6.1.0/src/toil/test/cwl/alwaysfails.cwl' [2024-04-30T11:57:55-1200] [MainThread] [I] [toil.cwl.cwltoil] Importing input files... [2024-04-30T11:57:55-1200] [MainThread] [I] [toil.cwl.cwltoil] Importing tool-associated files... [2024-04-30T11:57:55-1200] [MainThread] [I] [toil.cwl.cwltoil] Creating root job [2024-04-30T11:57:55-1200] [MainThread] [I] [toil.cwl.cwltoil] Starting workflow [2024-04-30T11:57:55-1200] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. Traceback (most recent call last): File "/build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/.local/bin/toil-cwl-runner", line 33, in sys.exit(load_entry_point('toil', 'console_scripts', 'toil-cwl-runner')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/cwl/cwltoil.py", line 3870, in main outobj = toil.start(wf1) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'CWLJob' alwaysfails.cwl kind-CWLJob/instance-xlgawzkh v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. __________________ UserDefinedJobArgTypeTest.testJobFunction ___________________ [gw0] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/userDefinedJobArgTypeTest.py", line 39, in testJobFunction Job.Runner.startToil(Job.wrapJobFn(jobFunction, 0, Foo()), self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'jobFunction' kind-jobFunction/instance-k7z98fhe v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.userDefinedJobArgTypeTest.UserDefinedJobArgTypeTest:testJobFunction (Apr 30 2024 11:57:57:801610 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.userDefinedJobArgTypeTest.UserDefinedJobArgTypeTest.testJobFunction ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.userDefinedJobArgTypeTest.UserDefinedJobArgTypeTest.testJobFunction ______________________ JobFileStoreTest.testJobFileStore _______________________ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobFileStoreTest.py", line 77, in testJobFileStore self._testJobFileStore(retryCount=0, badWorker=0.0, stringNo=5, stringLength=1000000) File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobFileStoreTest.py", line 69, in _testJobFileStore Job.Runner.startToil(Job.wrapJobFn(fileTestJob, [], File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'fileTestJob' kind-fileTestJob/instance-l6cy5hj3 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.jobFileStoreTest.JobFileStoreTest:testJobFileStore (Apr 30 2024 11:57:41:065779 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.jobFileStoreTest.JobFileStoreTest.testJobFileStore ... DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': None, 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.jobFileStoreTest.JobFileStoreTest-jobstore-4jzdvl3f', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 1.0, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.jobFileStoreTest.JobFileStoreTest-jobstore-4jzdvl3f'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '999d0e2e-3bda-4627-a393-6e83f70f69bf' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toilwf-e989d841cc8e51d7ab24ce7e050f76b6 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 2995607952. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 2995607952 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.jobFileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-cbf09272386643b58b79a02567096faf/stream for result of 'fileTestJob' TemporaryID(6e12a535-fd9e-449e-9c77-fa65748756e6) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job 'fileTestJob' kind-fileTestJob/instance-l6cy5hj3 v0 DEBUG toil.job:job.py:1299 New job version: 'fileTestJob' kind-fileTestJob/instance-l6cy5hj3 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '999d0e2e-3bda-4627-a393-6e83f70f69bf', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.jobFileStoreTest.JobFileStoreTest-jobstore-4jzdvl3f', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'onSuccess', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 1.0, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} DEBUG toil.realtimeLogger:realtimeLogger.py:162 Real-time logging disabled DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-fileTestJob/instance-l6cy5hj3, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-fileTestJob/instance-l6cy5hj3', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job 'fileTestJob' kind-fileTestJob/instance-l6cy5hj3 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 2.0004966259002686 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.002636432647705078 seconds DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 2995607952... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 2995607952 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-e989d841cc8e51d7ab24ce7e050f76b6/deferred/func_zeq0o87 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-e989d841cc8e51d7ab24ce7e050f76b6/deferred/func_zeq0o87 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toilwf-e989d841cc8e51d7ab24ce7e050f76b6 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-e989d841cc8e51d7ab24ce7e050f76b6 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.03613471984863281 seconds. INFO toil.test:__init__.py:120 Tore down toil.test.src.jobFileStoreTest.JobFileStoreTest.testJobFileStore ___________________ JobFileStoreTest.testNonCachingFileStore ___________________ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobFileStoreTest.py", line 44, in testNonCachingFileStore workflow.start(Job.wrapJobFn(simpleFileStoreJob)) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'simpleFileStoreJob' kind-simpleFileStoreJob/instance-3lrkpxe4 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.jobFileStoreTest.JobFileStoreTest:testNonCachingFileStore (Apr 30 2024 11:58:03:760645 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.jobFileStoreTest.JobFileStoreTest.testNonCachingFileStore ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.jobFileStoreTest.JobFileStoreTest.testNonCachingFileStore _________________ MessageBusTest.test_restart_without_bus_path _________________ [gw0] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/busTest.py", line 127, in test_restart_without_bus_path toil.start(root) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'failing_job_fn' kind-failing_job_fn/instance-c8zjexpm v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.busTest.MessageBusTest:test_restart_without_bus_path (Apr 30 2024 11:58:04:597003 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.busTest.MessageBusTest.test_restart_without_bus_path ... DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'charset_normalizer', 'docker', 'urllib3', 'requests', 'botocore', 'asyncio', 'boto3', 'boto', 'bcdocs', 'debugWorkflow', 'websocket', 'concurrent', 'dill'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': None, 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.busTest.MessageBusTest-jobstore-_4l9_w00', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'never', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': '/tmp/toil-test-src.busTest.MessageBusTest-test_restart_without_bus_path-tempDir-_ddccxec/bus_holder/messagebus.txt', 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.busTest.MessageBusTest-jobstore-_4l9_w00'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '8d00d71a-51ea-4a00-ada9-454dd3bb3431' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toilwf-5fcef97eba485bb987a99d044ad24782 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3028686000. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3028686000 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.busTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-f80bdd61e3d24100be1fa2397a8eeca6/stream for result of 'failing_job_fn' TemporaryID(1db30ade-6cfd-4fd7-82d2-6b434a72a4bd) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job 'failing_job_fn' kind-failing_job_fn/instance-c8zjexpm v0 DEBUG toil.job:job.py:1299 New job version: 'failing_job_fn' kind-failing_job_fn/instance-c8zjexpm v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '8d00d71a-51ea-4a00-ada9-454dd3bb3431', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.busTest.MessageBusTest-jobstore-_4l9_w00', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'never', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': '/tmp/toil-test-src.busTest.MessageBusTest-test_restart_without_bus_path-tempDir-_ddccxec/bus_holder/messagebus.txt', 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} DEBUG toil.realtimeLogger:realtimeLogger.py:162 Real-time logging disabled DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-failing_job_fn/instance-c8zjexpm, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-failing_job_fn/instance-c8zjexpm', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job 'failing_job_fn' kind-failing_job_fn/instance-c8zjexpm v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9963672161102295 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.007565498352050781 seconds DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3028686000... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3028686000 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-5fcef97eba485bb987a99d044ad24782/deferred/funcaeuz9dsj DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-5fcef97eba485bb987a99d044ad24782/deferred/funcaeuz9dsj DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toilwf-5fcef97eba485bb987a99d044ad24782 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-5fcef97eba485bb987a99d044ad24782 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.02837824821472168 seconds. INFO toil.test:__init__.py:120 Tore down toil.test.src.busTest.MessageBusTest.test_restart_without_bus_path ___________________ CheckpointTest.testCheckpointNotRetried ____________________ [gw0] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/checkpointTest.py", line 33, in testCheckpointNotRetried Job.Runner.startToil(CheckRetryCount(numFailuresBeforeSuccess=1), options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'CheckRetryCount' kind-CheckRetryCount/instance-hwrusor2 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.checkpointTest.CheckpointTest:testCheckpointNotRetried (Apr 30 2024 11:58:06:964099 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.checkpointTest.CheckpointTest.testCheckpointNotRetried ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.checkpointTest.CheckpointTest.testCheckpointNotRetried ____________________ JobEncapsulationTest.testEncapsulation ____________________ [gw0] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobEncapsulationTest.py", line 43, in testEncapsulation Job.Runner.startToil(a, options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'EncapsulatedJob' a-encap kind-EncapsulatedJob/instance-outx3y19 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.jobEncapsulationTest.JobEncapsulationTest:testEncapsulation (Apr 30 2024 11:58:12:679319 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.jobEncapsulationTest.JobEncapsulationTest.testEncapsulation ... INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.jobEncapsulationTest.JobEncapsulationTest.testEncapsulation ________________________ ResumabilityTest.test_chaining ________________________ [gw3] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/resumabilityTest.py", line 71, in test_chaining Job.Runner.startToil(root, options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'chaining_parent' kind-chaining_parent/instance-vy7_sf0a v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.resumabilityTest.ResumabilityTest:test_chaining (Apr 30 2024 11:58:14:291612 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.resumabilityTest.ResumabilityTest.test_chaining ... DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.resumabilityTest.ResumabilityTest-test_chaining-wnie76ca/log.txt'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'docker', 'debugWorkflow', 'botocore', 'dill', 'concurrent', 'bcdocs', 'requests', 'boto', 'charset_normalizer', 'boto3', 'urllib3', 'websocket', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.resumabilityTest.ResumabilityTest-test_chaining-wnie76ca/log.txt', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.resumabilityTest.ResumabilityTest-jobstore-jn_8_uk6', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.resumabilityTest.ResumabilityTest-jobstore-jn_8_uk6'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'ffea9c78-50fe-4f47-9bc8-9afcfcecc208' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toilwf-df1462c998c95f78af345b078b351d01 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 2995369040. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 2995369040 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.resumabilityTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-e69c032122f34b11921555d1ecc30b49/stream for result of 'chaining_parent' TemporaryID(9f87c974-6fe2-4548-9105-c3c7e9f16811) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job 'chaining_parent' kind-chaining_parent/instance-vy7_sf0a v0 DEBUG toil.job:job.py:1299 New job version: 'chaining_parent' kind-chaining_parent/instance-vy7_sf0a v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'ffea9c78-50fe-4f47-9bc8-9afcfcecc208', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.resumabilityTest.ResumabilityTest-jobstore-jn_8_uk6', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'onSuccess', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} DEBUG toil.realtimeLogger:realtimeLogger.py:162 Real-time logging disabled DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-chaining_parent/instance-vy7_sf0a, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-chaining_parent/instance-vy7_sf0a', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job 'chaining_parent' kind-chaining_parent/instance-vy7_sf0a v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9950146675109863 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.4966127872467041 seconds DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 2995369040... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 2995369040 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-df1462c998c95f78af345b078b351d01/deferred/funcucgxamax DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-df1462c998c95f78af345b078b351d01/deferred/funcucgxamax DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toilwf-df1462c998c95f78af345b078b351d01 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-df1462c998c95f78af345b078b351d01 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.03446149826049805 seconds. INFO toil.test:__init__.py:120 Tore down toil.test.src.resumabilityTest.ResumabilityTest.test_chaining ________________________ UtilsTest.testRestartAttribute ________________________ [gw2] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/utilsTest.py", line 399, in testRestartAttribute self.assertFalse(os.path.exists(self.toilDir)) File "/usr/lib/python3.12/unittest/case.py", line 721, in assertFalse raise self.failureException(msg) AssertionError: True is not false ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.utils.utilsTest.UtilsTest:testRestartAttribute (Apr 30 2024 11:57:58:265460 PDT) ----------------------------- Captured stdout call ----------------------------- No sort file specified. Generating one automatically called: fileToSort.txt. Sorting existing file: fileToSort.txt Sorting existing file: fileToSort.txt ----------------------------- Captured stderr call ----------------------------- [2024-04-30T11:58:02-1200] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. [2024-04-30T11:58:02-1200] [MainThread] [I] [toil.leader] Issued job 'setup' kind-setup/instance-ao4u6v1j v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 572.2 Mi, cores: 1, accelerators: [], preemptible: False [2024-04-30T11:58:04-1200] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run User defined signal 1 [2024-04-30T11:58:05-1200] [Thread-1 (daddy)] [E] [toil.batchSystems.singleMachine] Got exit code 138 (indicating failure) from job _toil_worker setup file:/tmp/toil-test-utils.utilsTest.UtilsTest-testRestartAttribute-epgmfvqs/jobstore kind-setup/instance-ao4u6v1j. [2024-04-30T11:58:05-1200] [MainThread] [W] [toil.leader] Job failed with exit value 138: 'setup' kind-setup/instance-ao4u6v1j v1 Exit reason: None [2024-04-30T11:58:05-1200] [MainThread] [W] [toil.leader] No log file is present, despite job failing: 'setup' kind-setup/instance-ao4u6v1j v1 [2024-04-30T11:58:05-1200] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'setup' kind-setup/instance-ao4u6v1j v1 with ID kind-setup/instance-ao4u6v1j to 1 [2024-04-30T11:58:05-1200] [MainThread] [W] [toil.job] We have increased the default memory of the failed job 'setup' kind-setup/instance-ao4u6v1j v1 to 2147483648 bytes Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/build/reproducible-path/toil-6.1.0/src/toil/test/sort/restart_sort.py", line 264, in main() File "/build/reproducible-path/toil-6.1.0/src/toil/test/sort/restart_sort.py", line 253, in main sortedFileID = workflow.start(Job.wrapJobFn(setup, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'setup' kind-setup/instance-ao4u6v1j v2 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. [2024-04-30T11:58:15-1200] [MainThread] [C] [toil.jobStores.abstractJobStore] Repairing job: kind-setup/instance-ao4u6v1j [2024-04-30T11:58:15-1200] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/build/reproducible-path/toil-6.1.0/src/toil/test/sort/sort.py", line 260, in main() File "/build/reproducible-path/toil-6.1.0/src/toil/test/sort/sort.py", line 255, in main sortedFileID = workflow.restart() ^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 952, in restart return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'setup' kind-setup/instance-ao4u6v1j v3 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. ____________________________ testFetchJobStoreFiles ____________________________ [gw0] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 340, in from_call result: Optional[TResult] = func() ^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 240, in lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 513, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 139, in _multicall raise exception.with_traceback(exception.__traceback__) File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 122, in _multicall teardown.throw(exception) # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/threadexception.py", line 87, in pytest_runtest_call yield from thread_exception_runtest_hook() File "/usr/lib/python3/dist-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook yield File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 122, in _multicall teardown.throw(exception) # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_call yield from unraisable_exception_runtest_hook() File "/usr/lib/python3/dist-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook yield File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 122, in _multicall teardown.throw(exception) # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/logging.py", line 849, in pytest_runtest_call yield from self._runtest_for(item, "call") File "/usr/lib/python3/dist-packages/_pytest/logging.py", line 832, in _runtest_for yield File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 122, in _multicall teardown.throw(exception) # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/capture.py", line 883, in pytest_runtest_call return (yield) ^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 122, in _multicall teardown.throw(exception) # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/skipping.py", line 256, in pytest_runtest_call return (yield) ^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 103, in _multicall res = hook_impl.function(*args) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 182, in pytest_runtest_call raise e File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 172, in pytest_runtest_call item.runtest() File "/usr/lib/python3/dist-packages/_pytest/python.py", line 1779, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 513, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 139, in _multicall raise exception.with_traceback(exception.__traceback__) File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 103, in _multicall res = hook_impl.function(*args) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/python.py", line 195, in pytest_pyfunc_call result = testfunction(**testargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/toilDebugTest.py", line 118, in testFetchJobStoreFiles job_store_dir = workflow_debug_jobstore() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/toilDebugTest.py", line 32, in workflow_debug_jobstore subprocess.check_call( File "/usr/lib/python3.12/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['python3.11', '/build/reproducible-path/toil-6.1.0/src/toil/test/utils/ABCWorkflowDebug/debugWorkflow.py', '/tmp/tmpju5e3cpw/toilWorkflowRun']' returned non-zero exit status 1. ----------------------------- Captured stderr call ----------------------------- [2024-04-30T11:58:18-1200] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. Traceback (most recent call last): File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/ABCWorkflowDebug/debugWorkflow.py", line 181, in toil.start(job0) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'initialize_jobs' kind-initialize_jobs/instance-xibcedfw v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. _________________________ UtilsTest.testUnicodeSupport _________________________ [gw2] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/utilsTest.py", line 287, in testUnicodeSupport Job.Runner.startToil(Job.wrapFn(printUnicodeCharacter), options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'printUnicodeCharacter' kind-printUnicodeCharacter/instance-gy2822x5 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.utils.utilsTest.UtilsTest:testUnicodeSupport (Apr 30 2024 11:58:18:910117 PDT) ------------------------------ Captured log call ------------------------------- DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'charset_normalizer', 'debugWorkflow', 'asyncio', 'concurrent', 'requests', 'urllib3', 'botocore', 'bcdocs', 'websocket', 'boto', 'boto3', 'docker', 'dill'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'charset_normalizer', 'debugWorkflow', 'asyncio', 'concurrent', 'requests', 'urllib3', 'botocore', 'bcdocs', 'websocket', 'boto', 'boto3', 'docker', 'dill'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'debug', 'logFile': None, 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-utils.utilsTest.UtilsTest-jobstore-f6g4t6b_', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-utils.utilsTest.UtilsTest-jobstore-f6g4t6b_'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'f5b011ee-f450-41c2-a93e-e57e13e1373b' DEBUG toil.common:common.py:740 Obtained node ID 9d1f8bcc-f705-4f67-84fb-c68d8c6c626b from file /proc/sys/kernel/random/boot_id DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toilwf-1de6220358df5a8c9a90a43d20467c7f DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3032818240. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3032818240 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.utils.utilsTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-1cedd0415dc54d9fbb31343f00a1969a/stream for result of 'printUnicodeCharacter' TemporaryID(039334b4-6a02-4b8d-a7fd-2f25e48f701b) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job 'printUnicodeCharacter' kind-printUnicodeCharacter/instance-gy2822x5 v0 DEBUG toil.job:job.py:1299 New job version: 'printUnicodeCharacter' kind-printUnicodeCharacter/instance-gy2822x5 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'f5b011ee-f450-41c2-a93e-e57e13e1373b', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-utils.utilsTest.UtilsTest-jobstore-f6g4t6b_', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'debug', 'workflowAttemptNumber': 0} DEBUG toil.realtimeLogger:realtimeLogger.py:162 Real-time logging disabled DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-printUnicodeCharacter/instance-gy2822x5, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-printUnicodeCharacter/instance-gy2822x5', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job 'printUnicodeCharacter' kind-printUnicodeCharacter/instance-gy2822x5 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.998370885848999 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.014365196228027344 seconds DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3032818240... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3032818240 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-1de6220358df5a8c9a90a43d20467c7f/deferred/funce6z_x4p2 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-1de6220358df5a8c9a90a43d20467c7f/deferred/funce6z_x4p2 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toilwf-1de6220358df5a8c9a90a43d20467c7f DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-1de6220358df5a8c9a90a43d20467c7f DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.031705379486083984 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-utils.utilsTest.UtilsTest-jobstore-f6g4t6b_) INFO toil.test:__init__.py:120 Tore down toil.test.utils.utilsTest.UtilsTest.testUnicodeSupport _________________________ PathIndexingPromiseTest.test _________________________ [gw1] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/promisesTest.py", line 69, in test self.assertEqual(Job.Runner.startToil(root, options), ('b', 43, 3)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'd' kind-d/instance-abgbpnc_ v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.promisesTest.PathIndexingPromiseTest:test (Apr 30 2024 11:58:32:762890 PDT) ------------------------------ Captured log call ------------------------------- INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.promisesTest.PathIndexingPromiseTest.test ________________________ HelloWorldTest.testHelloWorld _________________________ [gw2] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/helloWorldTest.py", line 23, in testHelloWorld Job.Runner.startToil(HelloWorld(), options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 564, in _processReadyJob self._runJobSuccessors(job_id) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 452, in _runJobSuccessors self.issueJobs(successors) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 936, in issueJobs self.issueJob(job) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'FollowOn' kind-FollowOn/instance-s9tt759m v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.helloWorldTest.HelloWorldTest:testHelloWorld (Apr 30 2024 11:58:26:487958 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.helloWorldTest.HelloWorldTest.testHelloWorld ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.leader:leader.py:923 Issued job 'HelloWorld' kind-HelloWorld/instance-egkvpcnw v1 with job batch system ID: 1 and disk: 2.8 Gi, memory: 97.7 Ki, cores: 1, accelerators: [], preemptible: False INFO toil.leader:leader.py:1004 1 jobs are running, 0 jobs are issued and waiting to run INFO toil.leader:leader.py:923 Issued job 'childFn' kind-childFn/instance-equcsjqr v1 with job batch system ID: 2 and disk: 2.8 Gi, memory: 976.6 Ki, cores: 1, accelerators: [], preemptible: False INFO toil.test:__init__.py:120 Tore down toil.test.src.helloWorldTest.HelloWorldTest.testHelloWorld ______________________ CachedUnpicklingJobStoreTest.test _______________________ [gw2] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/promisesTest.py", line 28, in test Job.Runner.startToil(root, options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'parent' kind-parent/instance-c891j0wp v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.promisesTest.CachedUnpicklingJobStoreTest:test (Apr 30 2024 11:58:41:971586 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.promisesTest.CachedUnpicklingJobStoreTest.test ... INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.promisesTest.CachedUnpicklingJobStoreTest.test _______________________ ChainedIndexedPromisesTest.test ________________________ [gw2] linux -- Python 3.12.3 /usr/bin/python3.12 Traceback (most recent call last): File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/usr/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/promisesTest.py", line 45, in test self.assertEqual(Job.Runner.startToil(root, options), 42) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'a' kind-a/instance-m2y62qv8 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.promisesTest.ChainedIndexedPromisesTest:test (Apr 30 2024 11:58:44:945417 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.promisesTest.ChainedIndexedPromisesTest.test ... INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.promisesTest.ChainedIndexedPromisesTest.test =========================== short test summary info ============================ FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testDeleteLocalFile FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCachHitFileFromJobStore FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile FAILED src/toil/test/src/jobTest.py::JobTest::testDAGConsistency - toil.batch... FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testSimultaneousReadsUncachedStream FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testToilIsNotBroken FAILED src/toil/test/src/jobTest.py::JobTest::testStatic2 - toil.batchSystems... FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testToilIsNotBroken FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteLocalFileToJobStore FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteNonLocalFileToJobStore FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions FAILED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testCachingFileStore FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedToilWF FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testPrintJobLog - Asserti... FAILED src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunction FAILED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStore FAILED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testNonCachingFileStore FAILED src/toil/test/src/busTest.py::MessageBusTest::test_restart_without_bus_path FAILED src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointNotRetried FAILED src/toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testEncapsulation FAILED src/toil/test/src/resumabilityTest.py::ResumabilityTest::test_chaining FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testRestartAttribute - As... FAILED src/toil/test/utils/toilDebugTest.py::testFetchJobStoreFiles - subproc... FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testUnicodeSupport - toil... FAILED src/toil/test/src/promisesTest.py::PathIndexingPromiseTest::test - toi... FAILED src/toil/test/src/helloWorldTest.py::HelloWorldTest::testHelloWorld - ... FAILED src/toil/test/src/promisesTest.py::CachedUnpicklingJobStoreTest::test FAILED src/toil/test/src/promisesTest.py::ChainedIndexedPromisesTest::test - ... =========== 34 failed, 132 passed, 349 skipped in 147.26s (0:02:27) ============ E: pybuild pybuild:389: test: plugin custom failed with: exit code=1: HOME=/build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil python3.12 setup.py develop --user && PYTHONPATH=/build/reproducible-path/toil-6.1.0/src:$PYTHONPATH PATH=/build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.12_toil/.local/bin/:$PATH TOIL_TEST_QUICK=True TOIL_SKIP_DOCKER=True python3.12 -m pytest -n auto --dist loadscope -vv -W ignore --ignore src/toil/test/lib/aws/test_s3.py --ignore src/toil/test/provisioners/aws/awsProvisionerTest.py --ignore src/toil/test/wdl/wdltoil_test.py --ignore src/toil/test/cwl/cwlTest.py --ignore src/toil/test/src/promisedRequirementTest.py --ignore src/toil/test/lib/test_ec2.py --ignore src/toil/test/batchSystems/batchSystemTest.py --ignore src/toil/test/lib/aws/test_iam.py -k "not (test_bioconda or test_run_conformance or testImportFtpFile or ToilWdlIntegrationTest or SortTest or testCwlexample or testVirtualEnv or ToilDocumentationTest or test_cwl_toil_kill or testImportReadFileCompatibility or CleanWorkDirTest or DeferredFunctionTest or WdlToilTest or test_cactus_integration)" /build/reproducible-path/toil-6.1.0/src/toil/test I: pybuild base:311: HOME=/build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil python3.11 setup.py develop --user && PYTHONPATH=/build/reproducible-path/toil-6.1.0/src:$PYTHONPATH PATH=/build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/.local/bin/:$PATH TOIL_TEST_QUICK=True TOIL_SKIP_DOCKER=True python3.11 -m pytest -n auto --dist loadscope -vv -W ignore --ignore src/toil/test/lib/aws/test_s3.py --ignore src/toil/test/provisioners/aws/awsProvisionerTest.py --ignore src/toil/test/wdl/wdltoil_test.py --ignore src/toil/test/cwl/cwlTest.py --ignore src/toil/test/src/promisedRequirementTest.py --ignore src/toil/test/lib/test_ec2.py --ignore src/toil/test/batchSystems/batchSystemTest.py --ignore src/toil/test/lib/aws/test_iam.py -k "not (test_bioconda or test_run_conformance or testImportFtpFile or ToilWdlIntegrationTest or SortTest or testCwlexample or testVirtualEnv or ToilDocumentationTest or test_cwl_toil_kill or testImportReadFileCompatibility or CleanWorkDirTest or DeferredFunctionTest or WdlToilTest or test_cactus_integration)" /build/reproducible-path/toil-6.1.0/src/toil/test running develop /usr/lib/python3/dist-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated. !! ******************************************************************************** Please avoid running ``setup.py`` and ``easy_install``. Instead, use pypa/build, pypa/installer or other standards-based tools. See https://github.com/pypa/setuptools/issues/917 for details. ******************************************************************************** !! easy_install.initialize_options(self) /usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated. !! ******************************************************************************** Please avoid running ``setup.py`` directly. Instead, use pypa/build, pypa/installer or other standards-based tools. See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details. ******************************************************************************** !! self.initialize_options() running egg_info writing src/toil.egg-info/PKG-INFO writing dependency_links to src/toil.egg-info/dependency_links.txt writing entry points to src/toil.egg-info/entry_points.txt writing requirements to src/toil.egg-info/requires.txt writing top-level names to src/toil.egg-info/top_level.txt reading manifest file 'src/toil.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' writing manifest file 'src/toil.egg-info/SOURCES.txt' running build_ext Creating /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/.local/lib/python3.11/site-packages/toil.egg-link (link to src) Adding toil 6.1.0 to easy-install.pth file Installing _toil_contained_executor script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/.local/bin Installing _toil_mesos_executor script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/.local/bin Installing _toil_worker script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/.local/bin Installing cwltoil script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/.local/bin Installing toil script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/.local/bin Installing toil-cwl-runner script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/.local/bin Installing toil-wdl-runner script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/.local/bin Installing toil-wes-cwl-runner script to /build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/.local/bin Installed /build/reproducible-path/toil-6.1.0/src Processing dependencies for toil==6.1.0 Searching for urllib3==1.26.18 Best match: urllib3 1.26.18 Adding urllib3 1.26.18 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for typing-extensions==4.10.0 Best match: typing-extensions 4.10.0 Adding typing-extensions 4.10.0 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for ruamel.yaml==0.18.6 Best match: ruamel.yaml 0.18.6 Adding ruamel.yaml 0.18.6 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for requests==2.31.0 Best match: requests 2.31.0 Adding requests 2.31.0 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for PyYAML==6.0.1 Best match: PyYAML 6.0.1 Adding PyYAML 6.0.1 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for pytz==2024.1 Best match: pytz 2024.1 Adding pytz 2024.1 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for python-dateutil==2.9.0 Best match: python-dateutil 2.9.0 Adding python-dateutil 2.9.0 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for psutil==5.9.8 Best match: psutil 5.9.8 Adding psutil 5.9.8 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for enlighten==1.11.2 Best match: enlighten 1.11.2 Adding enlighten 1.11.2 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for docker==6.1.3 Best match: docker 6.1.3 Adding docker 6.1.3 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for dill==0.3.8 Best match: dill 0.3.8 Adding dill 0.3.8 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for ConfigArgParse==1.7 Best match: ConfigArgParse 1.7 Adding ConfigArgParse 1.7 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for Pypubsub==4.0.3 Best match: Pypubsub 4.0.3 Adding Pypubsub 4.0.3 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for ruamel.yaml.clib==0.2.8 Best match: ruamel.yaml.clib 0.2.8 Adding ruamel.yaml.clib 0.2.8 to easy-install.pth file Using /usr/lib/python3/dist-packages Searching for six==1.16.0 Best match: six 1.16.0 Adding six 1.16.0 to easy-install.pth file Using /usr/lib/python3/dist-packages Finished processing dependencies for toil==6.1.0 ============================= test session starts ============================== platform linux -- Python 3.11.9, pytest-8.1.2, pluggy-1.5.0 -- /usr/bin/python3.11 cachedir: .pytest_cache rootdir: /build/reproducible-path/toil-6.1.0 configfile: setup.cfg plugins: xdist-3.5.0 created: 4/4 workers 4 workers [515 items] scheduling tests via LoadScopeScheduling src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testBatchCreate src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testBatchCreate src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testBatchCreate src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testBatchCreate [gw3] [ 0%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testBatchCreate src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testChildLoadingEquality [gw0] [ 0%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testBatchCreate src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testChildLoadingEquality [gw3] [ 0%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testChildLoadingEquality [gw2] [ 0%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testBatchCreate src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testCleanCache [gw0] [ 0%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testChildLoadingEquality src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testCleanCache src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testChildLoadingEquality [gw3] [ 1%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testCleanCache src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testConfigEquality [gw0] [ 1%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testCleanCache src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testConfigEquality [gw3] [ 1%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testConfigEquality [gw2] [ 1%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testChildLoadingEquality src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testDestructionIdempotence [gw3] [ 1%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testDestructionIdempotence src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testCleanCache src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testDestructionOfCorruptedJobStore [gw0] [ 2%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testConfigEquality src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testDestructionIdempotence [gw0] [ 2%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testDestructionIdempotence [gw2] [ 2%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testCleanCache src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testConfigEquality [gw3] [ 2%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testDestructionOfCorruptedJobStore src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testDestructionOfCorruptedJobStore [gw2] [ 2%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testConfigEquality [gw0] [ 3%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testDestructionOfCorruptedJobStore src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testEmptyFileStoreIDIsReadable src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testDestructionIdempotence [gw2] [ 3%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testDestructionIdempotence src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testEmptyFileStoreIDIsReadable src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testDestructionOfCorruptedJobStore [gw3] [ 3%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testEmptyFileStoreIDIsReadable [gw2] [ 3%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testDestructionOfCorruptedJobStore src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testFileDeletion src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testEmptyFileStoreIDIsReadable [gw2] [ 3%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testEmptyFileStoreIDIsReadable [gw0] [ 4%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testEmptyFileStoreIDIsReadable src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testEncrypted src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testFileDeletion [gw3] [ 4%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testFileDeletion [gw0] [ 4%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testEncrypted src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testFileDeletion src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testGrowingAndShrinkingJob [gw2] [ 4%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testFileDeletion src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testGrowingAndShrinkingJob [gw2] [ 4%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testGrowingAndShrinkingJob src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testImportHttpFile [gw0] [ 5%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testFileDeletion src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testGrowingAndShrinkingJob [gw0] [ 5%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testGrowingAndShrinkingJob [gw2] [ 5%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testImportHttpFile [gw3] [ 5%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testGrowingAndShrinkingJob src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testInitialState src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testImportHttpFile src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testImportHttpFile [gw0] [ 5%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testImportHttpFile [gw2] [ 6%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testInitialState src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testInlinedFiles src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testInitialState [gw3] [ 6%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testImportHttpFile [gw2] [ 6%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testInlinedFiles src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testJobCreation src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testInitialState [gw0] [ 6%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testInitialState src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testInlinedFiles [gw2] [ 6%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testJobCreation src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testJobDeletions [gw0] [ 6%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testInlinedFiles [gw2] [ 7%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testJobDeletions src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testJobCreation [gw3] [ 7%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testInitialState src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testJobLoadEquality [gw2] [ 7%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testJobLoadEquality src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testJobCreation src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testLargeFile [gw2] [ 7%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testLargeFile [gw0] [ 7%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testJobCreation src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testJobDeletions src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testMultiThreadImportFile [gw3] [ 8%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testJobCreation src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testJobDeletions [gw2] [ 8%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testMultiThreadImportFile src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testMultipartUploads [gw2] [ 8%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testMultipartUploads [gw0] [ 8%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testJobDeletions src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testJobLoadEquality src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testOverlargeJob [gw0] [ 8%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testJobLoadEquality src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testLargeFile [gw3] [ 9%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testJobDeletions src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testJobLoadEquality [gw2] [ 9%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testOverlargeJob src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testPartialReadFromStream [gw2] [ 9%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testPartialReadFromStream src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testPerJobFiles [gw0] [ 9%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testLargeFile src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testMultiThreadImportFile [gw0] [ 9%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testMultiThreadImportFile [gw2] [ 10%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testPerJobFiles src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testPersistantFilesToDelete [gw2] [ 10%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testPersistantFilesToDelete src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testReadWriteFileStreamTextMode [gw2] [ 10%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testReadWriteFileStreamTextMode src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testReadWriteSharedFilesTextMode [gw3] [ 10%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testJobLoadEquality src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testLargeFile [gw2] [ 10%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testReadWriteSharedFilesTextMode src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testMultipartUploads [gw0] [ 11%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testMultipartUploads src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testSDBDomainsDeletedOnFailedJobstoreBucketCreation [gw3] [ 11%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testLargeFile src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testMultipartUploads [gw2] [ 11%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testSDBDomainsDeletedOnFailedJobstoreBucketCreation src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testOverlargeJob [gw0] [ 11%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testOverlargeJob src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testSharedFiles [gw2] [ 11%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testSharedFiles src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testStatsAndLogging src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testPartialReadFromStream [gw0] [ 12%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testPartialReadFromStream [gw3] [ 12%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testMultipartUploads src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testPartialReadFromStream [gw2] [ 12%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testStatsAndLogging src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testUpdateBehavior src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testPerJobFiles [gw0] [ 12%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testPerJobFiles [gw2] [ 12%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testUpdateBehavior src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testWriteLogFiles src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testPersistantFilesToDelete [gw0] [ 13%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testPersistantFilesToDelete [gw2] [ 13%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testWriteLogFiles src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testZeroLengthFiles [gw2] [ 13%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::testZeroLengthFiles src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw3] [ 13%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testPartialReadFromStream src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testPerJobFiles [gw2] [ 13%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testReadWriteFileStreamTextMode [gw0] [ 13%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testReadWriteFileStreamTextMode [gw3] [ 14%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testPerJobFiles [gw2] [ 14%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testPersistantFilesToDelete [gw3] [ 14%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testPersistantFilesToDelete src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testReadWriteFileStreamTextMode [gw2] [ 14%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testReadWriteSharedFilesTextMode [gw0] [ 14%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testReadWriteSharedFilesTextMode src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testSDBDomainsDeletedOnFailedJobstoreBucketCreation src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw0] [ 15%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testSDBDomainsDeletedOnFailedJobstoreBucketCreation [gw2] [ 15%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw2] [ 15%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testSharedFiles [gw0] [ 15%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testSharedFiles src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testStatsAndLogging [gw3] [ 15%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testReadWriteFileStreamTextMode src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testReadWriteSharedFilesTextMode [gw2] [ 16%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw2] [ 16%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw0] [ 16%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testStatsAndLogging src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testUpdateBehavior [gw0] [ 16%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testUpdateBehavior src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw2] [ 16%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testWriteLogFiles [gw0] [ 17%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testWriteLogFiles src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testZeroLengthFiles [gw3] [ 17%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testReadWriteSharedFilesTextMode src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testSharedFiles [gw3] [ 17%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testSharedFiles [gw2] [ 17%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw0] [ 17%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::testZeroLengthFiles src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testStatsAndLogging [gw3] [ 18%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testStatsAndLogging src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testUpdateBehavior [gw2] [ 18%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw0] [ 18%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw2] [ 18%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw2] [ 18%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw3] [ 19%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testUpdateBehavior src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testWriteLogFiles [gw3] [ 19%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testWriteLogFiles src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testZeroLengthFiles [gw0] [ 19%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py [gw0] [ 19%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw0] [ 19%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw2] [ 20%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::AWSJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py [gw3] [ 20%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::testZeroLengthFiles src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw3] [ 20%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw3] [ 20%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py [gw3] [ 20%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw3] [ 20%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw0] [ 21%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw0] [ 21%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw0] [ 21%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw3] [ 21%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw3] [ 21%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw0] [ 22%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py [gw0] [ 22%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean [gw2] [ 22%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_FORGO [gw2] [ 22%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_FORGO src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_None [gw2] [ 22%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_None src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw3] [ 23%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw3] [ 23%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw0] [ 23%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py [gw3] [ 23%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_RM [gw2] [ 23%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_RM [gw0] [ 24%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_STOP src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw2] [ 24%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CRx_STOP src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw0] [ 24%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw3] [ 24%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_FORGO [gw2] [ 24%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_FORGO src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_None [gw2] [ 25%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_None src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_RM [gw2] [ 25%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_RM src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw3] [ 25%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_STOP [gw3] [ 25%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw2] [ 25%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_CxD_STOP src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_FORGO [gw2] [ 26%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_FORGO [gw0] [ 26%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::EncryptedAWSJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_None [gw2] [ 26%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_None src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_RM [gw2] [ 26%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_RM src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_STOP [gw2] [ 26%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_Cxx_STOP src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testAsyncWriteWithCaching src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_FORGO [gw2] [ 26%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_FORGO [gw3] [ 27%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::GoogleJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_None src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testAsyncWriteWithCaching [gw2] [ 27%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_None src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_RM [gw3] [ 27%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testAsyncWriteWithCaching src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testCacheEvictionFailCase [gw2] [ 27%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_RM [gw0] [ 27%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testAsyncWriteWithCaching src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_STOP [gw2] [ 28%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xRx_STOP src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_FORGO src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testCacheEvictionFailCase [gw3] [ 28%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testCacheEvictionFailCase src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testCacheEvictionPartialEvict [gw3] [ 28%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testCacheEvictionPartialEvict src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testCacheEvictionTotalEvict [gw0] [ 28%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testCacheEvictionFailCase [gw2] [ 28%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_FORGO src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_None src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testCacheEvictionPartialEvict [gw3] [ 29%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testCacheEvictionTotalEvict src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testControlledFailedWorkerRetry [gw3] [ 29%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testControlledFailedWorkerRetry [gw2] [ 29%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_None src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testDeleteLocalFile [gw0] [ 29%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testCacheEvictionPartialEvict [gw3] [ 29%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testDeleteLocalFile [gw1] [ 30%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testBatchCreate src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testChildLoadingEquality src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testExtremeCacheSetup src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testCacheEvictionTotalEvict src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_RM [gw3] [ 30%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testExtremeCacheSetup src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testFileStoreExportFile [gw2] [ 30%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_RM [gw0] [ 30%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testCacheEvictionTotalEvict [gw3] [ 30%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testFileStoreExportFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testControlledFailedWorkerRetry src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testFileStoreLogging src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_STOP [gw3] [ 31%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testFileStoreLogging src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testFileStoreOperations [gw0] [ 31%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testControlledFailedWorkerRetry [gw2] [ 31%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxD_STOP [gw3] [ 31%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testFileStoreOperations src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testDeleteLocalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testMultipleJobsReadSameCacheHitGlobalFile src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_FORGO [gw3] [ 31%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testMultipleJobsReadSameCacheHitGlobalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testMultipleJobsReadSameCacheMissGlobalFile [gw2] [ 32%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_FORGO [gw3] [ 32%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testMultipleJobsReadSameCacheMissGlobalFile src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_None src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadCachHitFileFromJobStore [gw3] [ 32%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadCachHitFileFromJobStore [gw2] [ 32%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_None src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile [gw3] [ 32%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_RM src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile [gw3] [ 33%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile [gw2] [ 33%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_RM src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadWriteFileStreamTextMode src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_STOP [gw3] [ 33%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReadWriteFileStreamTextMode src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testRemoveLocalImmutablyReadFile [gw2] [ 33%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerClean_xxx_STOP [gw3] [ 33%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testRemoveLocalImmutablyReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testRemoveLocalMutablyReadFile src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs [gw3] [ 33%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testRemoveLocalMutablyReadFile [gw1] [ 34%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testChildLoadingEquality src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReturnFileSizes [gw2] [ 34%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testCleanCache [gw3] [ 34%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReturnFileSizes src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs_Demux src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReturnFileSizesWithBadWorker [gw3] [ 34%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testReturnFileSizesWithBadWorker [gw1] [ 34%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testCleanCache [gw2] [ 35%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs_Demux src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testSimultaneousReadsUncachedStream [gw3] [ 35%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testSimultaneousReadsUncachedStream src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs_Demux_Stream src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testToilIsNotBroken src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testConfigEquality [gw3] [ 35%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testToilIsNotBroken [gw2] [ 35%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs_Demux_Stream src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteExportFileCompatibility src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs_Stream [gw3] [ 35%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteExportFileCompatibility src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteLocalFileToJobStore [gw2] [ 36%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerLogs_Stream [gw3] [ 36%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteNonLocalFileToJobStore src/toil/test/lib/dockerTest.py::DockerTest::testDockerPipeChain [gw3] [ 36%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteNonLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteReadGlobalFilePermissions [gw2] [ 36%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerPipeChain src/toil/test/lib/dockerTest.py::DockerTest::testDockerPipeChainErrorDetection [gw3] [ 36%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithAwsJobStore::testWriteReadGlobalFilePermissions src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testAsyncWriteWithCaching [gw3] [ 37%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testAsyncWriteWithCaching [gw2] [ 37%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testDockerPipeChainErrorDetection src/toil/test/lib/dockerTest.py::DockerTest::testNonCachingDockerChain src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testCacheEvictionFailCase [gw3] [ 37%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testCacheEvictionFailCase src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testCacheEvictionPartialEvict [gw2] [ 37%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testNonCachingDockerChain [gw3] [ 37%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testCacheEvictionPartialEvict src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testCacheEvictionTotalEvict [gw3] [ 38%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testCacheEvictionTotalEvict src/toil/test/lib/dockerTest.py::DockerTest::testNonCachingDockerChainErrorDetection [gw2] [ 38%] SKIPPED src/toil/test/lib/dockerTest.py::DockerTest::testNonCachingDockerChainErrorDetection src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_many_all_exist src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testControlledFailedWorkerRetry [gw3] [ 38%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testControlledFailedWorkerRetry src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testDeleteLocalFile [gw1] [ 38%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testConfigEquality src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testDestructionIdempotence [gw3] [ 38%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testDeleteLocalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testExtremeCacheSetup [gw3] [ 39%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testExtremeCacheSetup src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testFileStoreExportFile [gw3] [ 39%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testFileStoreExportFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testFileStoreLogging [gw3] [ 39%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testFileStoreLogging src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testFileStoreOperations [gw3] [ 39%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testFileStoreOperations src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testMultipleJobsReadSameCacheHitGlobalFile [gw3] [ 39%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testMultipleJobsReadSameCacheHitGlobalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testMultipleJobsReadSameCacheMissGlobalFile [gw3] [ 40%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testMultipleJobsReadSameCacheMissGlobalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadCachHitFileFromJobStore [gw3] [ 40%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadCachHitFileFromJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile [gw1] [ 40%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testDestructionIdempotence [gw3] [ 40%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testDestructionOfCorruptedJobStore [gw1] [ 40%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testDestructionOfCorruptedJobStore [gw3] [ 40%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadWriteFileStreamTextMode src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testEmptyFileStoreIDIsReadable [gw3] [ 41%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReadWriteFileStreamTextMode src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testRemoveLocalImmutablyReadFile [gw3] [ 41%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testRemoveLocalImmutablyReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testRemoveLocalMutablyReadFile [gw3] [ 41%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testRemoveLocalMutablyReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReturnFileSizes [gw3] [ 41%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReturnFileSizes src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReturnFileSizesWithBadWorker [gw1] [ 41%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testEmptyFileStoreIDIsReadable [gw3] [ 42%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testReturnFileSizesWithBadWorker src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testFileDeletion src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testSimultaneousReadsUncachedStream [gw3] [ 42%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testSimultaneousReadsUncachedStream [gw1] [ 42%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testFileDeletion src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testToilIsNotBroken src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testGrowingAndShrinkingJob [gw3] [ 42%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testToilIsNotBroken src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteExportFileCompatibility [gw3] [ 42%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteExportFileCompatibility src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteLocalFileToJobStore [gw3] [ 43%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteNonLocalFileToJobStore [gw2] [ 43%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_many_all_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_one_exists [gw3] [ 43%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteNonLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteReadGlobalFilePermissions [gw3] [ 43%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithGoogleJobStore::testWriteReadGlobalFilePermissions src/toil/test/utils/utilsTest.py::UtilsTest::testAWSProvisionerUtils [gw3] [ 43%] SKIPPED src/toil/test/utils/utilsTest.py::UtilsTest::testAWSProvisionerUtils src/toil/test/utils/utilsTest.py::UtilsTest::testGetPIDStatus [gw2] [ 44%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_one_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_one_not_exists [gw1] [ 44%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testGrowingAndShrinkingJob src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testImportHttpFile [gw2] [ 44%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_one_not_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_sacct_raises_job_exists [gw2] [ 44%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_sacct_raises_job_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_sacct_raises_job_not_exists [gw2] [ 44%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_sacct_raises_job_not_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_some_exists [gw2] [ 45%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_coalesce_job_exit_codes_some_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_many_all_exist [gw2] [ 45%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_many_all_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_many_none_exist [gw1] [ 45%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testImportHttpFile src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testInitialState [gw2] [ 45%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_many_none_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_many_some_exist [gw2] [ 45%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_many_some_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_one_exists [gw2] [ 46%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_one_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_one_not_exists [gw2] [ 46%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromSacct_one_not_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_many_all_exist [gw2] [ 46%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_many_all_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_many_none_exist [gw2] [ 46%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_many_none_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_many_some_exist [gw2] [ 46%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_many_some_exist src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_one_exists [gw2] [ 46%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_one_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_one_not_exists [gw2] [ 47%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobDetailsFromScontrol_one_not_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_job_exists [gw2] [ 47%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_job_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_job_not_exists [gw2] [ 47%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_job_not_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_sacct_raises_job_exists [gw2] [ 47%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_sacct_raises_job_exists src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_sacct_raises_job_not_exists [gw2] [ 47%] PASSED src/toil/test/batchSystems/test_slurm.py::SlurmTest::test_getJobExitCode_sacct_raises_job_not_exists src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testApplianceParser [gw2] [ 48%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testApplianceParser src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadGoogleRepo [gw2] [ 48%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadGoogleRepo src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayRepo [gw2] [ 48%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayRepo src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayRepoNTag [gw0] [ 48%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testDeleteLocalFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testExtremeCacheSetup [gw2] [ 48%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayRepoNTag src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayTag [gw0] [ 49%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testExtremeCacheSetup [gw2] [ 49%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayTag src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testFileStoreExportFile src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBroadDockerRepo [gw2] [ 49%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBroadDockerRepo [gw0] [ 49%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testFileStoreExportFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testFileStoreLogging src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBroadDockerRepoBadTag [gw2] [ 49%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testBroadDockerRepoBadTag [gw0] [ 50%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testFileStoreLogging src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testGoogleRepo src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testFileStoreOperations [gw2] [ 50%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testGoogleRepo src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testNonexistentRepo [gw0] [ 50%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testFileStoreOperations src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testMultipleJobsReadSameCacheHitGlobalFile [gw2] [ 50%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testNonexistentRepo src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testOfficialUbuntuRepo [gw0] [ 50%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testMultipleJobsReadSameCacheHitGlobalFile [gw2] [ 51%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testOfficialUbuntuRepo src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testMultipleJobsReadSameCacheMissGlobalFile [gw0] [ 51%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testMultipleJobsReadSameCacheMissGlobalFile src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testToilQuayRepo [gw2] [ 51%] SKIPPED src/toil/test/src/dockerCheckTest.py::DockerCheckTest::testToilQuayRepo src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCachHitFileFromJobStore src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testBetaInertia [gw2] [ 51%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testBetaInertia src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testMaxNodes [gw2] [ 51%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testMaxNodes src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testMinNodes [gw2] [ 52%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testMinNodes src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testNoLaunchingIfDeltaAlreadyMet [gw2] [ 52%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testNoLaunchingIfDeltaAlreadyMet src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testPreemptibleDeficitIsSet [gw2] [ 52%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testPreemptibleDeficitIsSet src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testPreemptibleDeficitResponse [gw2] [ 52%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testPreemptibleDeficitResponse src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testRounding [gw2] [ 52%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testRounding src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_large [gw2] [ 53%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_large src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_observed [gw2] [ 53%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_observed src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_small [gw2] [ 53%] PASSED src/toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_small src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_and_cancel_workflows [gw2] [ 53%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_and_cancel_workflows src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_https_url [gw2] [ 53%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_https_url src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_inputs_zip [gw2] [ 53%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_inputs_zip src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_manifest_and_inputs_zip [gw2] [ 54%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_manifest_and_inputs_zip src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_manifest_zip [gw2] [ 54%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_manifest_zip src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_multi_file_zip [gw2] [ 54%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_multi_file_zip src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_no_params_zip [gw2] [ 54%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_no_params_zip src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_relative_url [gw2] [ 54%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_relative_url src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_relative_url_no_attachments_fails [gw2] [ 55%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_relative_url_no_attachments_fails src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_single_file_zip [gw2] [ 55%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_single_file_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_and_cancel_workflows [gw2] [ 55%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_and_cancel_workflows src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_https_url [gw2] [ 55%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_https_url src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_inputs_zip [gw2] [ 55%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_inputs_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_manifest_and_inputs_zip [gw2] [ 56%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_manifest_and_inputs_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_manifest_zip [gw2] [ 56%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_manifest_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_multi_file_zip [gw2] [ 56%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_multi_file_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_no_params_zip [gw2] [ 56%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_no_params_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_relative_url [gw2] [ 56%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_relative_url src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_relative_url_no_attachments_fails [gw2] [ 57%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_relative_url_no_attachments_fails src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_single_file_zip [gw2] [ 57%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_single_file_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_and_cancel_workflows [gw2] [ 57%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_and_cancel_workflows src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_https_url [gw2] [ 57%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_https_url src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_inputs_zip [gw2] [ 57%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_inputs_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_manifest_and_inputs_zip [gw2] [ 58%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_manifest_and_inputs_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_manifest_zip [gw2] [ 58%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_manifest_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_multi_file_zip [gw2] [ 58%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_multi_file_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_no_params_zip [gw2] [ 58%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_no_params_zip src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_relative_url [gw2] [ 58%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_relative_url src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_relative_url_no_attachments_fails [gw2] [ 59%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_relative_url_no_attachments_fails src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_single_file_zip [gw2] [ 59%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_single_file_zip src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testAddingInitialNode [gw2] [ 59%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testAddingInitialNode src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testHighTargetTime [gw0] [ 59%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCachHitFileFromJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile [gw2] [ 59%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testHighTargetTime src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testJobTooLargeForAllNodes [gw2] [ 60%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testJobTooLargeForAllNodes src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testLongRunningJobs [gw0] [ 60%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile [gw2] [ 60%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testLongRunningJobs src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testLowTargetTime [gw2] [ 60%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testLowTargetTime src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testPackingOneShape [gw2] [ 60%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testPackingOneShape src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testPathologicalCase [gw2] [ 60%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testPathologicalCase src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testSorting [gw2] [ 61%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testSorting src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testZeroResourceJobs [gw2] [ 61%] PASSED src/toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testZeroResourceJobs src/toil/test/src/jobTest.py::JobTest::testDAGConsistency [gw0] [ 61%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode [gw2] [ 61%] FAILED src/toil/test/src/jobTest.py::JobTest::testDAGConsistency src/toil/test/src/jobTest.py::JobTest::testDeadlockDetection [gw0] [ 61%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testRemoveLocalImmutablyReadFile [gw0] [ 62%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testRemoveLocalImmutablyReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testRemoveLocalMutablyReadFile [gw0] [ 62%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testRemoveLocalMutablyReadFile src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReturnFileSizes [gw0] [ 62%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReturnFileSizes src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReturnFileSizesWithBadWorker [gw0] [ 62%] SKIPPED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReturnFileSizesWithBadWorker src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testSimultaneousReadsUncachedStream [gw3] [ 62%] PASSED src/toil/test/utils/utilsTest.py::UtilsTest::testGetPIDStatus src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedCWLWF [gw3] [ 63%] SKIPPED src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedCWLWF src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedToilWF [gw0] [ 63%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testSimultaneousReadsUncachedStream src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testToilIsNotBroken [gw0] [ 63%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testToilIsNotBroken src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility [gw2] [ 63%] PASSED src/toil/test/src/jobTest.py::JobTest::testDeadlockDetection src/toil/test/src/jobTest.py::JobTest::testEvaluatingRandomDAG [gw2] [ 63%] SKIPPED src/toil/test/src/jobTest.py::JobTest::testEvaluatingRandomDAG src/toil/test/src/jobTest.py::JobTest::testNewCheckpointIsLeafVertexNonRootCase [gw2] [ 64%] SKIPPED src/toil/test/src/jobTest.py::JobTest::testNewCheckpointIsLeafVertexNonRootCase src/toil/test/src/jobTest.py::JobTest::testNewCheckpointIsLeafVertexRootCase [gw2] [ 64%] SKIPPED src/toil/test/src/jobTest.py::JobTest::testNewCheckpointIsLeafVertexRootCase src/toil/test/src/jobTest.py::JobTest::testSiblingDAGConsistency [gw2] [ 64%] SKIPPED src/toil/test/src/jobTest.py::JobTest::testSiblingDAGConsistency src/toil/test/src/jobTest.py::JobTest::testStatic [gw2] [ 64%] SKIPPED src/toil/test/src/jobTest.py::JobTest::testStatic src/toil/test/src/jobTest.py::JobTest::testStatic2 [gw0] [ 64%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteLocalFileToJobStore [gw2] [ 65%] FAILED src/toil/test/src/jobTest.py::JobTest::testStatic2 src/toil/test/src/jobTest.py::JobTest::testTrivialDAGConsistency [gw2] [ 65%] SKIPPED src/toil/test/src/jobTest.py::JobTest::testTrivialDAGConsistency src/toil/test/src/miscTests.py::MiscTests::testGetSizeOfDirectoryWorks [gw2] [ 65%] SKIPPED src/toil/test/src/miscTests.py::MiscTests::testGetSizeOfDirectoryWorks src/toil/test/src/miscTests.py::MiscTests::testIDStability [gw2] [ 65%] PASSED src/toil/test/src/miscTests.py::MiscTests::testIDStability src/toil/test/src/miscTests.py::MiscTests::test_atomic_context_error [gw2] [ 65%] PASSED src/toil/test/src/miscTests.py::MiscTests::test_atomic_context_error src/toil/test/src/miscTests.py::MiscTests::test_atomic_context_ok [gw2] [ 66%] PASSED src/toil/test/src/miscTests.py::MiscTests::test_atomic_context_ok src/toil/test/src/miscTests.py::MiscTests::test_atomic_install [gw2] [ 66%] PASSED src/toil/test/src/miscTests.py::MiscTests::test_atomic_install src/toil/test/src/miscTests.py::MiscTests::test_atomic_install_dev [gw2] [ 66%] PASSED src/toil/test/src/miscTests.py::MiscTests::test_atomic_install_dev src/toil/test/src/miscTests.py::MiscTests::test_call_command_err [gw2] [ 66%] PASSED src/toil/test/src/miscTests.py::MiscTests::test_call_command_err src/toil/test/src/miscTests.py::MiscTests::test_call_command_ok [gw2] [ 66%] PASSED src/toil/test/src/miscTests.py::MiscTests::test_call_command_ok src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testFileStoreLogging [gw2] [ 66%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testFileStoreLogging src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testFileStoreOperations [gw2] [ 67%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testFileStoreOperations src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode [gw0] [ 67%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteNonLocalFileToJobStore [gw2] [ 67%] FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testToilIsNotBroken [gw1] [ 67%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testInitialState src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testJobCreation [gw1] [ 67%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testJobCreation src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testJobDeletions [gw2] [ 68%] FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testToilIsNotBroken src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility [gw0] [ 68%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteNonLocalFileToJobStore src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions [gw2] [ 68%] FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions [gw0] [ 68%] FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testFileStoreLogging [gw0] [ 68%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testFileStoreLogging src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testFileStoreOperations [gw0] [ 69%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testFileStoreOperations src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testReadWriteFileStreamTextMode [gw0] [ 69%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testReadWriteFileStreamTextMode src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testToilIsNotBroken [gw0] [ 69%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testToilIsNotBroken src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testWriteExportFileCompatibility [gw0] [ 69%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testWriteExportFileCompatibility src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testWriteReadGlobalFilePermissions [gw0] [ 69%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testWriteReadGlobalFilePermissions src/toil/test/src/jobServiceTest.py::JobServiceTest::testService [gw0] [ 70%] SKIPPED src/toil/test/src/jobServiceTest.py::JobServiceTest::testService src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceDeadlock [gw0] [ 70%] SKIPPED src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceDeadlock src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceParallelRecursive [gw0] [ 70%] SKIPPED src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceParallelRecursive src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceRecursive [gw0] [ 70%] SKIPPED src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceRecursive src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceSerialization [gw0] [ 70%] SKIPPED src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceSerialization src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceWithCheckpoints [gw0] [ 71%] SKIPPED src/toil/test/src/jobServiceTest.py::JobServiceTest::testServiceWithCheckpoints src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testService [gw0] [ 71%] SKIPPED src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testService src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceDeadlock [gw0] [ 71%] SKIPPED src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceDeadlock src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceParallelRecursive [gw0] [ 71%] SKIPPED src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceParallelRecursive src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceRecursive [gw0] [ 71%] SKIPPED src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceRecursive src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceSerialization [gw0] [ 72%] SKIPPED src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceSerialization src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceWithCheckpoints [gw0] [ 72%] SKIPPED src/toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceWithCheckpoints src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_build_tag [gw0] [ 72%] PASSED src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_build_tag src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_build_tag_with_tags [gw0] [ 72%] PASSED src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_build_tag_with_tags src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_empty_aws_tags [gw0] [ 72%] PASSED src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_empty_aws_tags src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_incorrect_json_emoji [gw0] [ 73%] PASSED src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_incorrect_json_emoji src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_incorrect_json_object [gw0] [ 73%] PASSED src/toil/test/lib/aws/test_utils.py::TagGenerationTest::test_incorrect_json_object src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testDeferralWithConcurrentEncapsulation [gw0] [ 73%] SKIPPED src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testDeferralWithConcurrentEncapsulation src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testDeferralWithFailureAndEncapsulation [gw0] [ 73%] SKIPPED src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testDeferralWithFailureAndEncapsulation src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testRestart [gw0] [ 73%] SKIPPED src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testRestart src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testSplitRootPackages [gw0] [ 73%] SKIPPED src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testSplitRootPackages src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testUserTypesInJobFunctionArgs [gw0] [ 74%] SKIPPED src/toil/test/src/autoDeploymentTest.py::AutoDeploymentTest::testUserTypesInJobFunctionArgs src/toil/test/src/regularLogTest.py::RegularLogTest::testLogToMaster [gw0] [ 74%] SKIPPED src/toil/test/src/regularLogTest.py::RegularLogTest::testLogToMaster src/toil/test/src/regularLogTest.py::RegularLogTest::testMultipleLogToMaster [gw0] [ 74%] SKIPPED src/toil/test/src/regularLogTest.py::RegularLogTest::testMultipleLogToMaster src/toil/test/src/regularLogTest.py::RegularLogTest::testRegularLog [gw2] [ 74%] FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testFileStoreLogging [gw2] [ 74%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testFileStoreLogging src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testFileStoreOperations [gw2] [ 75%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testFileStoreOperations src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testReadWriteFileStreamTextMode [gw2] [ 75%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testReadWriteFileStreamTextMode src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testToilIsNotBroken [gw2] [ 75%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testToilIsNotBroken src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testWriteExportFileCompatibility [gw2] [ 75%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testWriteExportFileCompatibility src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testWriteReadGlobalFilePermissions [gw2] [ 75%] SKIPPED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testWriteReadGlobalFilePermissions src/toil/test/src/resourceTest.py::ResourceTest::testBuiltIn [gw2] [ 76%] PASSED src/toil/test/src/resourceTest.py::ResourceTest::testBuiltIn src/toil/test/src/resourceTest.py::ResourceTest::testNonPyStandAlone [gw2] [ 76%] PASSED src/toil/test/src/resourceTest.py::ResourceTest::testNonPyStandAlone src/toil/test/src/resourceTest.py::ResourceTest::testPackage [gw2] [ 76%] PASSED src/toil/test/src/resourceTest.py::ResourceTest::testPackage src/toil/test/src/resourceTest.py::ResourceTest::testStandAlone [gw2] [ 76%] PASSED src/toil/test/src/resourceTest.py::ResourceTest::testStandAlone src/toil/test/src/resourceTest.py::ResourceTest::testStandAloneInPackage [gw2] [ 76%] PASSED src/toil/test/src/resourceTest.py::ResourceTest::testStandAloneInPackage src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testCachingFileStore [gw2] [ 77%] FAILED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testCachingFileStore src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStore [gw3] [ 77%] FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedToilWF src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusSuccessfulCWLWF [gw3] [ 77%] SKIPPED src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusSuccessfulCWLWF src/toil/test/utils/utilsTest.py::UtilsTest::testMultipleJobsPerWorkerStats [gw3] [ 77%] SKIPPED src/toil/test/utils/utilsTest.py::UtilsTest::testMultipleJobsPerWorkerStats src/toil/test/utils/utilsTest.py::UtilsTest::testPrintJobLog [gw0] [ 77%] PASSED src/toil/test/src/regularLogTest.py::RegularLogTest::testRegularLog src/toil/test/src/regularLogTest.py::RegularLogTest::testWriteGzipLogs [gw0] [ 78%] SKIPPED src/toil/test/src/regularLogTest.py::RegularLogTest::testWriteGzipLogs src/toil/test/src/regularLogTest.py::RegularLogTest::testWriteLogs [gw0] [ 78%] PASSED src/toil/test/src/regularLogTest.py::RegularLogTest::testWriteLogs src/toil/test/src/miscTests.py::TestPanic::test_nested_panic [gw0] [ 78%] PASSED src/toil/test/src/miscTests.py::TestPanic::test_nested_panic src/toil/test/src/miscTests.py::TestPanic::test_panic [gw0] [ 78%] PASSED src/toil/test/src/miscTests.py::TestPanic::test_panic src/toil/test/src/miscTests.py::TestPanic::test_panic_by_hand [gw0] [ 78%] PASSED src/toil/test/src/miscTests.py::TestPanic::test_panic_by_hand src/toil/test/src/miscTests.py::TestPanic::test_panic_with_secondary [gw0] [ 79%] PASSED src/toil/test/src/miscTests.py::TestPanic::test_panic_with_secondary src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobClass [gw0] [ 79%] SKIPPED src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobClass src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobClassFromMain [gw3] [ 79%] FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testPrintJobLog src/toil/test/utils/utilsTest.py::UtilsTest::testRestartAttribute [gw0] [ 79%] PASSED src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobClassFromMain src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunction [gw0] [ 79%] FAILED src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunction src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunctionFromMain [gw0] [ 80%] PASSED src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunctionFromMain src/toil/test/lib/test_conversions.py::ConversionTest::test_convert [gw0] [ 80%] PASSED src/toil/test/lib/test_conversions.py::ConversionTest::test_convert src/toil/test/lib/test_conversions.py::ConversionTest::test_hms_duration_to_seconds [gw0] [ 80%] PASSED src/toil/test/lib/test_conversions.py::ConversionTest::test_hms_duration_to_seconds src/toil/test/lib/test_conversions.py::ConversionTest::test_human2bytes [gw0] [ 80%] PASSED src/toil/test/lib/test_conversions.py::ConversionTest::test_human2bytes src/toil/test/provisioners/clusterScalerTest.py::ScalerThreadTest::testClusterScaling [gw0] [ 80%] SKIPPED src/toil/test/provisioners/clusterScalerTest.py::ScalerThreadTest::testClusterScaling src/toil/test/provisioners/clusterScalerTest.py::ScalerThreadTest::testClusterScalingMultipleNodeTypes [gw0] [ 80%] SKIPPED src/toil/test/provisioners/clusterScalerTest.py::ScalerThreadTest::testClusterScalingMultipleNodeTypes src/toil/test/provisioners/clusterScalerTest.py::ScalerThreadTest::testClusterScalingWithPreemptibleJobs [gw0] [ 81%] SKIPPED src/toil/test/provisioners/clusterScalerTest.py::ScalerThreadTest::testClusterScalingWithPreemptibleJobs src/toil/test/server/serverTest.py::ToilWESServerBenchTest::test_get_service_info [gw0] [ 81%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerBenchTest::test_get_service_info src/toil/test/server/serverTest.py::ToilWESServerBenchTest::test_health [gw0] [ 81%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerBenchTest::test_health src/toil/test/server/serverTest.py::ToilWESServerBenchTest::test_home [gw0] [ 81%] SKIPPED src/toil/test/server/serverTest.py::ToilWESServerBenchTest::test_home src/toil/test/src/busTest.py::MessageBusTest::test_cross_thread_messaging [gw0] [ 81%] PASSED src/toil/test/src/busTest.py::MessageBusTest::test_cross_thread_messaging src/toil/test/src/busTest.py::MessageBusTest::test_enum_ints_in_file [gw0] [ 82%] PASSED src/toil/test/src/busTest.py::MessageBusTest::test_enum_ints_in_file src/toil/test/src/busTest.py::MessageBusTest::test_restart_without_bus_path [gw2] [ 82%] FAILED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStore src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStoreWithBadWorker [gw2] [ 82%] SKIPPED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStoreWithBadWorker src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testNonCachingFileStore [gw0] [ 82%] FAILED src/toil/test/src/busTest.py::MessageBusTest::test_restart_without_bus_path src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointNotRetried [gw2] [ 82%] FAILED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testNonCachingFileStore src/toil/test/src/importExportFileTest.py::ImportExportFileTest::test_basic_import_export [gw2] [ 83%] PASSED src/toil/test/src/importExportFileTest.py::ImportExportFileTest::test_basic_import_export src/toil/test/src/importExportFileTest.py::ImportExportFileTest::test_import_export_restart_false [gw0] [ 83%] FAILED src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointNotRetried src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointRetriedOnce [gw0] [ 83%] SKIPPED src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointRetriedOnce src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointedRestartSucceeds [gw0] [ 83%] SKIPPED src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointedRestartSucceeds src/toil/test/__init__.py::toil.test.make_tests [gw0] [ 83%] PASSED src/toil/test/__init__.py::toil.test.make_tests src/toil/test/__init__.py::toil.test.timeLimit [gw0] [ 84%] PASSED src/toil/test/__init__.py::toil.test.timeLimit src/toil/test/options/options.py::OptionsTest::test_caching_option_priority [gw0] [ 84%] PASSED src/toil/test/options/options.py::OptionsTest::test_caching_option_priority src/toil/test/options/options.py::OptionsTest::test_default_caching_slurm [gw0] [ 84%] PASSED src/toil/test/options/options.py::OptionsTest::test_default_caching_slurm src/toil/test/provisioners/gceProvisionerTest.py::GCEAutoscaleTest::testAutoScale [gw0] [ 84%] SKIPPED src/toil/test/provisioners/gceProvisionerTest.py::GCEAutoscaleTest::testAutoScale src/toil/test/provisioners/gceProvisionerTest.py::GCEAutoscaleTest::testSpotAutoScale [gw0] [ 84%] SKIPPED src/toil/test/provisioners/gceProvisionerTest.py::GCEAutoscaleTest::testSpotAutoScale src/toil/test/provisioners/gceProvisionerTest.py::GCEStaticAutoscaleTest::testAutoScale [gw0] [ 85%] SKIPPED src/toil/test/provisioners/gceProvisionerTest.py::GCEStaticAutoscaleTest::testAutoScale src/toil/test/provisioners/gceProvisionerTest.py::GCEStaticAutoscaleTest::testSpotAutoScale [gw0] [ 85%] SKIPPED src/toil/test/provisioners/gceProvisionerTest.py::GCEStaticAutoscaleTest::testSpotAutoScale src/toil/test/server/serverTest.py::AWSStateStoreTest::test_state_store [gw0] [ 85%] SKIPPED src/toil/test/server/serverTest.py::AWSStateStoreTest::test_state_store src/toil/test/server/serverTest.py::AWSStateStoreTest::test_state_store_paths [gw0] [ 85%] SKIPPED src/toil/test/server/serverTest.py::AWSStateStoreTest::test_state_store_paths src/toil/test/src/jobDescriptionTest.py::JobDescriptionTest::testJobDescription [gw0] [ 85%] PASSED src/toil/test/src/jobDescriptionTest.py::JobDescriptionTest::testJobDescription src/toil/test/src/jobDescriptionTest.py::JobDescriptionTest::testJobDescriptionSequencing [gw0] [ 86%] PASSED src/toil/test/src/jobDescriptionTest.py::JobDescriptionTest::testJobDescriptionSequencing src/toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testAddChildEncapsulate [gw0] [ 86%] PASSED src/toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testAddChildEncapsulate src/toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testEncapsulation [gw0] [ 86%] FAILED src/toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testEncapsulation src/toil/test/src/restartDAGTest.py::RestartDAGTest::testRestartedWorkflowSchedulesCorrectJobsOnFailedParent [gw2] [ 86%] PASSED src/toil/test/src/importExportFileTest.py::ImportExportFileTest::test_import_export_restart_false src/toil/test/src/importExportFileTest.py::ImportExportFileTest::test_import_export_restart_true [gw0] [ 86%] SKIPPED src/toil/test/src/restartDAGTest.py::RestartDAGTest::testRestartedWorkflowSchedulesCorrectJobsOnFailedParent src/toil/test/src/restartDAGTest.py::RestartDAGTest::testRestartedWorkflowSchedulesCorrectJobsOnKilledParent [gw2] [ 86%] SKIPPED src/toil/test/src/importExportFileTest.py::ImportExportFileTest::test_import_export_restart_true src/toil/test/src/toilContextManagerTest.py::ToilContextManagerTest::testContextManger [gw0] [ 87%] SKIPPED src/toil/test/src/restartDAGTest.py::RestartDAGTest::testRestartedWorkflowSchedulesCorrectJobsOnKilledParent src/toil/test/src/resumabilityTest.py::ResumabilityTest::test [gw2] [ 87%] SKIPPED src/toil/test/src/toilContextManagerTest.py::ToilContextManagerTest::testContextManger src/toil/test/src/toilContextManagerTest.py::ToilContextManagerTest::testExportAfterFailedExport [gw0] [ 87%] SKIPPED src/toil/test/src/resumabilityTest.py::ResumabilityTest::test src/toil/test/src/resumabilityTest.py::ResumabilityTest::test_chaining [gw2] [ 87%] SKIPPED src/toil/test/src/toilContextManagerTest.py::ToilContextManagerTest::testExportAfterFailedExport src/toil/test/src/toilContextManagerTest.py::ToilContextManagerTest::testNoContextManger [gw2] [ 87%] SKIPPED src/toil/test/src/toilContextManagerTest.py::ToilContextManagerTest::testNoContextManger src/toil/test/utils/toilDebugTest.py::testJobStoreContents [gw2] [ 88%] SKIPPED src/toil/test/utils/toilDebugTest.py::testJobStoreContents src/toil/test/utils/toilDebugTest.py::testFetchJobStoreFiles [gw3] [ 88%] FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testRestartAttribute src/toil/test/utils/utilsTest.py::UtilsTest::testUnicodeSupport [gw0] [ 88%] FAILED src/toil/test/src/resumabilityTest.py::ResumabilityTest::test_chaining src/toil/test/src/threadingTest.py::ThreadingTest::testGlobalMutexOrdering [gw3] [ 88%] FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testUnicodeSupport src/toil/test/utils/utilsTest.py::UtilsTest::testUtilsSort [gw3] [ 88%] SKIPPED src/toil/test/utils/utilsTest.py::UtilsTest::testUtilsSort src/toil/test/utils/utilsTest.py::UtilsTest::testUtilsStatsSort [gw3] [ 89%] SKIPPED src/toil/test/utils/utilsTest.py::UtilsTest::testUtilsStatsSort src/toil/test/utils/utilsTest.py::UtilsTest::test_config_functionality [gw1] [ 89%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testJobDeletions src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testJobLoadEquality [gw1] [ 89%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testJobLoadEquality src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testLargeFile [gw1] [ 89%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testLargeFile src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testMultipartUploads [gw1] [ 89%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testMultipartUploads src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPartialReadFromStream [gw1] [ 90%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPartialReadFromStream src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPerJobFiles [gw1] [ 90%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPerJobFiles src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPersistantFilesToDelete [gw1] [ 90%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPersistantFilesToDelete src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPreserveFileName [gw1] [ 90%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testPreserveFileName src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testReadWriteFileStreamTextMode [gw1] [ 90%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testReadWriteFileStreamTextMode src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testReadWriteSharedFilesTextMode [gw1] [ 91%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testReadWriteSharedFilesTextMode src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testSharedFiles [gw1] [ 91%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testSharedFiles src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testStatsAndLogging [gw2] [ 91%] FAILED src/toil/test/utils/toilDebugTest.py::testFetchJobStoreFiles src/toil/test/utils/toilDebugTest.py::DebugJobTest::test_print_job_info [gw1] [ 91%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testStatsAndLogging src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testUpdateBehavior [gw1] [ 91%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testUpdateBehavior src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testWriteLogFiles [gw1] [ 92%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testWriteLogFiles src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testZeroLengthFiles [gw1] [ 92%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::testZeroLengthFiles src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_file_link_imports [gw1] [ 92%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_file_link_imports src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_jobstore_does_not_leak_symlinks [gw1] [ 92%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_jobstore_does_not_leak_symlinks src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_jobstore_init_preserves_symlink_path [gw3] [ 92%] PASSED src/toil/test/utils/utilsTest.py::UtilsTest::test_config_functionality [gw1] [ 93%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_jobstore_init_preserves_symlink_path src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::InvalidAWSJobStoreTest::testInvalidJobStoreName [gw3] [ 93%] SKIPPED src/toil/test/jobStores/jobStoreTest.py::InvalidAWSJobStoreTest::testInvalidJobStoreName src/toil/test/lib/test_misc.py::UserNameAvailableTest::test_get_user_name [gw3] [ 93%] PASSED src/toil/test/lib/test_misc.py::UserNameAvailableTest::test_get_user_name src/toil/test/lib/test_misc.py::UserNameVeryBrokenTest::test_get_user_name [gw3] [ 93%] PASSED src/toil/test/lib/test_misc.py::UserNameVeryBrokenTest::test_get_user_name src/toil/test/mesos/MesosDataStructuresTest.py::DataStructuresTest::testJobQueue [gw1] [ 93%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw3] [ 93%] PASSED src/toil/test/mesos/MesosDataStructuresTest.py::DataStructuresTest::testJobQueue src/toil/test/provisioners/gceProvisionerTest.py::GCEAutoscaleTestMultipleNodeTypes::testAutoScale [gw3] [ 94%] SKIPPED src/toil/test/provisioners/gceProvisionerTest.py::GCEAutoscaleTestMultipleNodeTypes::testAutoScale [gw1] [ 94%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/provisioners/gceProvisionerTest.py::GCERestartTest::testAutoScaledCluster [gw3] [ 94%] SKIPPED src/toil/test/provisioners/gceProvisionerTest.py::GCERestartTest::testAutoScaledCluster src/toil/test/provisioners/provisionerTest.py::ProvisionerTest::test_node_type_parsing [gw3] [ 94%] PASSED src/toil/test/provisioners/provisionerTest.py::ProvisionerTest::test_node_type_parsing src/toil/test/server/serverTest.py::ToilServerUtilsTest::test_workflow_canceling_recovery [gw3] [ 94%] SKIPPED src/toil/test/server/serverTest.py::ToilServerUtilsTest::test_workflow_canceling_recovery src/toil/test/server/serverTest.py::FileStateStoreTest::test_state_store [gw3] [ 95%] SKIPPED src/toil/test/server/serverTest.py::FileStateStoreTest::test_state_store src/toil/test/server/serverTest.py::FileStateStoreURLTest::test_state_store [gw3] [ 95%] SKIPPED src/toil/test/server/serverTest.py::FileStateStoreURLTest::test_state_store src/toil/test/src/helloWorldTest.py::HelloWorldTest::testHelloWorld [gw1] [ 95%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw1] [ 95%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw1] [ 95%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw1] [ 96%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_activated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py [gw1] [ 96%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_one <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py [gw1] [ 96%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_onemib <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py [gw1] [ 96%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsize <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py [gw1] [ 96%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeminusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py [gw1] [ 97%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_partsizeplusone <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py [gw1] [ 97%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportExportFile__moveExports_deactivated__otherCls_filejobstoretest__size_zero <- src/toil/test/__init__.py src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py [gw1] [ 97%] PASSED src/toil/test/jobStores/jobStoreTest.py::FileJobStoreTest::test_testImportSharedFile__otherCls_filejobstoretest <- src/toil/test/__init__.py src/toil/test/src/promisesTest.py::PathIndexingPromiseTest::test [gw1] [ 97%] FAILED src/toil/test/src/promisesTest.py::PathIndexingPromiseTest::test src/toil/test/src/realtimeLoggerTest.py::RealtimeLoggerTest::testRealtimeLogger [gw2] [ 97%] PASSED src/toil/test/utils/toilDebugTest.py::DebugJobTest::test_print_job_info src/toil/test/utils/toilDebugTest.py::DebugJobTest::test_run_job [gw3] [ 98%] FAILED src/toil/test/src/helloWorldTest.py::HelloWorldTest::testHelloWorld src/toil/test/src/promisesTest.py::CachedUnpicklingJobStoreTest::test [gw1] [ 98%] PASSED src/toil/test/src/realtimeLoggerTest.py::RealtimeLoggerTest::testRealtimeLogger src/toil/test/src/systemTest.py::SystemTest::testAtomicityOfNonEmptyDirectoryRenames [gw3] [ 98%] FAILED src/toil/test/src/promisesTest.py::CachedUnpicklingJobStoreTest::test src/toil/test/src/promisesTest.py::ChainedIndexedPromisesTest::test [gw0] [ 98%] PASSED src/toil/test/src/threadingTest.py::ThreadingTest::testGlobalMutexOrdering src/toil/test/src/threadingTest.py::ThreadingTest::testLastProcessStanding [gw3] [ 98%] FAILED src/toil/test/src/promisesTest.py::ChainedIndexedPromisesTest::test [gw2] [ 99%] PASSED src/toil/test/utils/toilDebugTest.py::DebugJobTest::test_run_job src/toil/test/lib/test_misc.py::UserNameUnvailableTest::test_get_user_name [gw2] [ 99%] PASSED src/toil/test/lib/test_misc.py::UserNameUnvailableTest::test_get_user_name [gw1] [ 99%] PASSED src/toil/test/src/systemTest.py::SystemTest::testAtomicityOfNonEmptyDirectoryRenames src/toil/test/src/workerTest.py::WorkerTests::testNextChainable [gw1] [ 99%] PASSED src/toil/test/src/workerTest.py::WorkerTests::testNextChainable [gw0] [ 99%] PASSED src/toil/test/src/threadingTest.py::ThreadingTest::testLastProcessStanding src/toil/test/batchSystems/test_lsf_helper.py::LSFHelperTest::test_parse_mem_and_cmd_from_output [gw0] [100%] PASSED src/toil/test/batchSystems/test_lsf_helper.py::LSFHelperTest::test_parse_mem_and_cmd_from_output =================================== FAILURES =================================== ___________ CachingFileStoreTestWithFileJobStore.testDeleteLocalFile ___________ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 1229, in testDeleteLocalFile Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_deleteLocalFileFn' kind-_deleteLocalFileFn/instance-42_jgc34 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testDeleteLocalFile (Apr 30 2024 11:59:28:194356 PDT) ------------------------------ Captured log call ------------------------------- DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testDeleteLocalFile-epoexxi_/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testDeleteLocalFile-epoexxi_/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-cqig7xcz', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testDeleteLocalFile-epoexxi_', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-cqig7xcz'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '90df9497-55b3-4425-8e21-cb3cf29e5d8b' DEBUG toil.common:common.py:740 Obtained node ID 9d1f8bcc-f705-4f67-84fb-c68d8c6c626b from file /proc/sys/kernel/random/boot_id DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testDeleteLocalFile-epoexxi_/toilwf-994f4a9955ec57089ac41ed393adbb43 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3022555760. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3022555760 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-4d3b2dc5db6c4f4e939aecac63ade7c8/stream for result of '_deleteLocalFileFn' TemporaryID(17d56010-3949-4ebb-8a1c-f5e4aa03baf6) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_deleteLocalFileFn' kind-_deleteLocalFileFn/instance-42_jgc34 v0 DEBUG toil.job:job.py:1299 New job version: '_deleteLocalFileFn' kind-_deleteLocalFileFn/instance-42_jgc34 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '90df9497-55b3-4425-8e21-cb3cf29e5d8b', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-cqig7xcz', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testDeleteLocalFile-epoexxi_', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_deleteLocalFileFn/instance-42_jgc34, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_deleteLocalFileFn/instance-42_jgc34', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_deleteLocalFileFn' kind-_deleteLocalFileFn/instance-42_jgc34 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9981117248535156 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.0027751922607421875 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3022555760... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3022555760 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-994f4a9955ec57089ac41ed393adbb43/deferred/funcv1lbdui3 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-994f4a9955ec57089ac41ed393adbb43/deferred/funcv1lbdui3 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testDeleteLocalFile-epoexxi_/toilwf-994f4a9955ec57089ac41ed393adbb43 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-994f4a9955ec57089ac41ed393adbb43 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.023479461669921875 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-cqig7xcz) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testDeleteLocalFile _____ CachingFileStoreTestWithFileJobStore.testReadCachHitFileFromJobStore _____ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 805, in testReadCachHitFileFromJobStore Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-4urmswfs v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testReadCachHitFileFromJobStore (Apr 30 2024 11:59:31:893477 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadCachHitFileFromJobStore ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCachHitFileFromJobStore-zugn9k5j/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCachHitFileFromJobStore-zugn9k5j/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-2beeu31y', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCachHitFileFromJobStore-zugn9k5j', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-2beeu31y'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '649b69a7-8419-43b7-ac25-0fd16fe8f1e2' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCachHitFileFromJobStore-zugn9k5j/toilwf-930c68c4813f585a9042f746e3c3e6cd DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3003832464. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3003832464 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-f1854ca8abf940cebc7a0c5e86566ae6/stream for result of '_writeFileToJobStoreWithAsserts' TemporaryID(452737b7-2135-4884-9883-1bfc505281a7) v0 DEBUG toil.job:job.py:2624 Saving graph of 2 jobs, 2 non-service, 2 new DEBUG toil.job:job.py:2644 Processing job '_readFromJobStore' kind-_readFromJobStore/instance-0cay7aki v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-eca7eb196fbb4945806ceef6100c780f/stream for result of '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-4urmswfs v0 DEBUG toil.job:job.py:2644 Processing job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-4urmswfs v0 DEBUG toil.job:job.py:1299 New job version: '_readFromJobStore' kind-_readFromJobStore/instance-0cay7aki v1 DEBUG toil.job:job.py:1299 New job version: '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-4urmswfs v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '649b69a7-8419-43b7-ac25-0fd16fe8f1e2', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-2beeu31y', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCachHitFileFromJobStore-zugn9k5j', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_writeFileToJobStoreWithAsserts/instance-4urmswfs, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_writeFileToJobStoreWithAsserts/instance-4urmswfs', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-4urmswfs v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.998713493347168 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.006067991256713867 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3003832464... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3003832464 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-930c68c4813f585a9042f746e3c3e6cd/deferred/funcupcuc7m8 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-930c68c4813f585a9042f746e3c3e6cd/deferred/funcupcuc7m8 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCachHitFileFromJobStore-zugn9k5j/toilwf-930c68c4813f585a9042f746e3c3e6cd DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-930c68c4813f585a9042f746e3c3e6cd DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.01901531219482422 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-2beeu31y) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadCachHitFileFromJobStore _ CachingFileStoreTestWithFileJobStore.testReadCacheMissFileFromJobStoreWithCachingReadFile _ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 735, in testReadCacheMissFileFromJobStoreWithCachingReadFile self._testCacheMissFunction(cacheReadFile=True) File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 749, in _testCacheMissFunction Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-s62recl2 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testReadCacheMissFileFromJobStoreWithCachingReadFile (Apr 30 2024 11:59:34:827710 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadCacheMissFileFromJobStoreWithCachingReadFile ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithCachingReadFile-9m92fi7r/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithCachingReadFile-9m92fi7r/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-d6xk3tq4', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithCachingReadFile-9m92fi7r', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-d6xk3tq4'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '85375f7b-ab4d-4a9f-a0f6-c22d1acba2cb' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithCachingReadFile-9m92fi7r/toilwf-f07de31caf01589d984841b05327d365 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3003582832. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3003582832 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-f7d9f22bfa3d4fe0aa1a8d01501c786f/stream for result of '_writeFileToJobStoreWithAsserts' TemporaryID(68e58cf4-1f5e-46b5-a00a-65cb67d6601c) v0 DEBUG toil.job:job.py:2624 Saving graph of 2 jobs, 2 non-service, 2 new DEBUG toil.job:job.py:2644 Processing job '_readFromJobStore' kind-_readFromJobStore/instance-fxk2g71m v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-61d8d760bfb545649e1b3415512554d9/stream for result of '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-s62recl2 v0 DEBUG toil.job:job.py:2644 Processing job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-s62recl2 v0 DEBUG toil.job:job.py:1299 New job version: '_readFromJobStore' kind-_readFromJobStore/instance-fxk2g71m v1 DEBUG toil.job:job.py:1299 New job version: '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-s62recl2 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '85375f7b-ab4d-4a9f-a0f6-c22d1acba2cb', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-d6xk3tq4', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithCachingReadFile-9m92fi7r', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_writeFileToJobStoreWithAsserts/instance-s62recl2, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_writeFileToJobStoreWithAsserts/instance-s62recl2', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-s62recl2 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9963147640228271 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5025160312652588 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3003582832... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3003582832 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-f07de31caf01589d984841b05327d365/deferred/func6tfrxyko DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-f07de31caf01589d984841b05327d365/deferred/func6tfrxyko DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithCachingReadFile-9m92fi7r/toilwf-f07de31caf01589d984841b05327d365 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-f07de31caf01589d984841b05327d365 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.020892858505249023 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-d6xk3tq4) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadCacheMissFileFromJobStoreWithCachingReadFile _ CachingFileStoreTestWithFileJobStore.testReadCacheMissFileFromJobStoreWithoutCachingReadFile _ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 728, in testReadCacheMissFileFromJobStoreWithoutCachingReadFile self._testCacheMissFunction(cacheReadFile=False) File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 749, in _testCacheMissFunction Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-dwsfm1g1 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testReadCacheMissFileFromJobStoreWithoutCachingReadFile (Apr 30 2024 11:59:38:268817 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadCacheMissFileFromJobStoreWithoutCachingReadFile ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithoutCachingReadFile-14mq88hr/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithoutCachingReadFile-14mq88hr/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-ydpkbjlc', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithoutCachingReadFile-14mq88hr', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-ydpkbjlc'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '8f9abd94-9103-4032-accb-8514be781b67' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithoutCachingReadFile-14mq88hr/toilwf-ed63067ac4ac555c88502480f7e94681 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3003834000. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3003834000 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-803eaf8ff0544c109fda7866a2d2731d/stream for result of '_writeFileToJobStoreWithAsserts' TemporaryID(80142aaa-101b-4ace-b013-ec73d2a0ef86) v0 DEBUG toil.job:job.py:2624 Saving graph of 2 jobs, 2 non-service, 2 new DEBUG toil.job:job.py:2644 Processing job '_readFromJobStore' kind-_readFromJobStore/instance-7fps7d48 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-9b23e5fe652042fa81cd45211c9a7f82/stream for result of '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-dwsfm1g1 v0 DEBUG toil.job:job.py:2644 Processing job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-dwsfm1g1 v0 DEBUG toil.job:job.py:1299 New job version: '_readFromJobStore' kind-_readFromJobStore/instance-7fps7d48 v1 DEBUG toil.job:job.py:1299 New job version: '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-dwsfm1g1 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '8f9abd94-9103-4032-accb-8514be781b67', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-ydpkbjlc', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithoutCachingReadFile-14mq88hr', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_writeFileToJobStoreWithAsserts/instance-dwsfm1g1, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_writeFileToJobStoreWithAsserts/instance-dwsfm1g1', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-dwsfm1g1 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.996159315109253 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5016508102416992 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3003834000... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3003834000 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-ed63067ac4ac555c88502480f7e94681/deferred/func1g4dtguw DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-ed63067ac4ac555c88502480f7e94681/deferred/func1g4dtguw DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadCacheMissFileFromJobStoreWithoutCachingReadFile-14mq88hr/toilwf-ed63067ac4ac555c88502480f7e94681 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-ed63067ac4ac555c88502480f7e94681 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.028351783752441406 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-ydpkbjlc) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadCacheMissFileFromJobStoreWithoutCachingReadFile __________________________ JobTest.testDAGConsistency __________________________ [gw2] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobTest.py", line 145, in testDAGConsistency toil.start(i) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'parent' kind-parent/instance-7zc23698 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.jobTest.JobTest:testDAGConsistency (Apr 30 2024 11:59:41:710764 PDT) ------------------------------ Captured log call ------------------------------- DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'websocket', 'requests', 'boto3', 'urllib3', 'botocore', 'boto', 'docker', 'asyncio', 'concurrent', 'charset_normalizer', 'bcdocs', 'dill', 'debugWorkflow'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'debug', 'logFile': None, 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.jobTest.JobTest-testDAGConsistency-vwrfmod6/jobStore', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.jobTest.JobTest-testDAGConsistency-vwrfmod6/jobStore'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '07dfdcd2-8089-4121-b885-c43f6dfdf8f5' DEBUG toil.common:common.py:740 Obtained node ID 9d1f8bcc-f705-4f67-84fb-c68d8c6c626b from file /proc/sys/kernel/random/boot_id DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toilwf-e23c5c34666a50639fe62c012156a745 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3022966480. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3022966480 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.jobTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-28d0ab39745842e08f9a50a8be004b82/stream for result of 'parent' TemporaryID(c6f7924d-bf88-4bb4-9637-37e563fb0285) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job 'parent' kind-parent/instance-7zc23698 v0 DEBUG toil.job:job.py:1299 New job version: 'parent' kind-parent/instance-7zc23698 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '07dfdcd2-8089-4121-b885-c43f6dfdf8f5', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.jobTest.JobTest-testDAGConsistency-vwrfmod6/jobStore', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'debug', 'workflowAttemptNumber': 0} DEBUG toil.realtimeLogger:realtimeLogger.py:162 Real-time logging disabled DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-parent/instance-7zc23698, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-parent/instance-7zc23698', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job 'parent' kind-parent/instance-7zc23698 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9979619979858398 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.0025103092193603516 seconds DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3022966480... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3022966480 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-e23c5c34666a50639fe62c012156a745/deferred/funcpn1td_7n DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-e23c5c34666a50639fe62c012156a745/deferred/funcpn1td_7n DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toilwf-e23c5c34666a50639fe62c012156a745 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-e23c5c34666a50639fe62c012156a745 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.025555849075317383 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.jobTest.JobTest-testDAGConsistency-vwrfmod6/jobStore) INFO toil.test:__init__.py:120 Tore down toil.test.src.jobTest.JobTest.testDAGConsistency _____ CachingFileStoreTestWithFileJobStore.testReadWriteFileStreamTextMode _____ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 351, in testReadWriteFileStreamTextMode toil.start(A) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-7hd6gqoi v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testReadWriteFileStreamTextMode (Apr 30 2024 11:59:41:737126 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadWriteFileStreamTextMode ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-3wngalqc/logFile'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-3wngalqc/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-xjns1fvc', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-3wngalqc', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-xjns1fvc'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '4f2a40d0-55c4-4555-aca6-289021db629b' DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-3wngalqc/toilwf-1126e3e78e10523e94c0f9eba3a93d84 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 2993487632. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 2993487632 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-166f57e3eb7e42b8a83c444516a88808/stream for result of '_testReadWriteFileStreamTextMode' TemporaryID(a4de49c2-e722-47b2-9bf3-8bee390c706e) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-7hd6gqoi v0 DEBUG toil.job:job.py:1299 New job version: '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-7hd6gqoi v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '4f2a40d0-55c4-4555-aca6-289021db629b', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-xjns1fvc', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-3wngalqc', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_testReadWriteFileStreamTextMode/instance-7hd6gqoi, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_testReadWriteFileStreamTextMode/instance-7hd6gqoi', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-7hd6gqoi v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9983916282653809 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.4981374740600586 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 2993487632... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 2993487632 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-1126e3e78e10523e94c0f9eba3a93d84/deferred/funccb3wl5m9 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-1126e3e78e10523e94c0f9eba3a93d84/deferred/funccb3wl5m9 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-3wngalqc/toilwf-1126e3e78e10523e94c0f9eba3a93d84 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-1126e3e78e10523e94c0f9eba3a93d84 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.029599428176879883 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-xjns1fvc) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testReadWriteFileStreamTextMode ___ CachingFileStoreTestWithFileJobStore.testSimultaneousReadsUncachedStream ___ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 1302, in testSimultaneousReadsUncachedStream Job.Runner.startToil(parent, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testSimultaneousReadsUncachedStream (Apr 30 2024 11:59:45:249297 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testSimultaneousReadsUncachedStream ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testSimultaneousReadsUncachedStream-gsn2xvwk/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testSimultaneousReadsUncachedStream-gsn2xvwk/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-osgtn4if', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testSimultaneousReadsUncachedStream-gsn2xvwk', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': True, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-osgtn4if'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '69b5521f-efa7-4807-91a1-3bb3479a90b7' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testSimultaneousReadsUncachedStream-gsn2xvwk/toilwf-e7e7bd96e66759e6a038e0bf2fc4725b DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 2993183568. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 2993183568 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-e11ed45e219f46c88efd1cbfcfa1b880/stream for result of '_createUncachedFileStream' TemporaryID(6f78f69c-2731-4276-8fb0-17c402aa7cc0) v0 DEBUG toil.job:job.py:2624 Saving graph of 31 jobs, 31 non-service, 31 new DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-edrzupej v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-d3750bf587294f1e854a9ed836361977/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-hbu8lwg6 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-b27f0b9fb1804d6f9b16123a2d9585a9/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-0kqyizfi v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-153a738a3df84451bddb10411e265a4c/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-uxkb05gm v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-d48f4c1c5b7b448ab1c77449a9c8ad92/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-wcdlq_3a v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-ef84e71f645a4cea98e31f56d8c7073a/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-66csxx30 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-fb57d530bbb54809bf51383b32c7d9c3/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-zn_qg5dd v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-ee723e5911af47c79f89153f3cccb6a0/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-_3h21op3 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-2399f70306424d0b90fdd117063d577a/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-zgvsycaf v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-f1dee2e49d7b423eb56d813a2874b138/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-cuea8_mm v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-6709b5fcb26449fe887799c5800186ea/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-63w_b5mv v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-67d6254de8ab4879a34f25912913d696/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-oponezne v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-479e9b3378ea47e8aec0a02cc8002f33/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-zsvccz5s v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-34b62eaa80a740f1bb4300d8c537ebbc/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-0rysnt7r v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-a92d894efc844e3f9d44ea6ce0eafb5e/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-04pjfbna v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-bb61b436151249cd9f701e55976c0f95/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-c26nb9_f v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-8970ce9670934fed8fd4fb6f1a4444d9/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-ki66t04r v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-0cd8c95c898d47bbb17972e81601866d/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-g4nfy7e1 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-626fc7acd32343e3853fb137877e5377/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-gmter6ne v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-98260e2fdd324cff85baadf631d92138/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-2tgd702f v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-bef81e6e88334eac96478db9ff2ee9e0/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-mk80uhm2 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-8ff70b58d506455d866d96666ba53456/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-92fvzj6p v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-9b66706d6b24444a904e2deaa295c7ad/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-719e8lqo v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-5954d6923d654a7cb6506bb97aa2fdba/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-yxagk7on v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-beeffcec853941098e2d2395ef04af71/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-hwsojnw5 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-376aec7a22504898acf9d0a0d5866bd8/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-oulgvrcy v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-fa0faf320f4c49aaa4016c06b4e0eb9d/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-7rcwu009 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-e62b3491e9a04489ae965a96e4b9a533/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-7toayww3 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-c3c1c4eb417842c4a5570a176921db63/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-jnwpilei v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-5323d9ecfcf34013b05f4c67015c6a55/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_readFileWithDelay' kind-_readFileWithDelay/instance-od3s61i0 v0 DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-969bcc2a12004d4496ef5d881cef1847/stream for result of '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:2644 Processing job '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v0 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-edrzupej v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-hbu8lwg6 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-0kqyizfi v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-uxkb05gm v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-wcdlq_3a v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-66csxx30 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-zn_qg5dd v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-_3h21op3 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-zgvsycaf v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-cuea8_mm v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-63w_b5mv v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-oponezne v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-zsvccz5s v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-0rysnt7r v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-04pjfbna v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-c26nb9_f v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-ki66t04r v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-g4nfy7e1 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-gmter6ne v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-2tgd702f v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-mk80uhm2 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-92fvzj6p v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-719e8lqo v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-yxagk7on v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-hwsojnw5 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-oulgvrcy v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-7rcwu009 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-7toayww3 v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-jnwpilei v1 DEBUG toil.job:job.py:1299 New job version: '_readFileWithDelay' kind-_readFileWithDelay/instance-od3s61i0 v1 DEBUG toil.job:job.py:1299 New job version: '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '69b5521f-efa7-4807-91a1-3bb3479a90b7', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-osgtn4if', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testSimultaneousReadsUncachedStream-gsn2xvwk', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': True, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_createUncachedFileStream/instance-cmcwud17, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_createUncachedFileStream/instance-cmcwud17', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_createUncachedFileStream' kind-_createUncachedFileStream/instance-cmcwud17 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9971258640289307 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.49612927436828613 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 2993183568... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 2993183568 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-e7e7bd96e66759e6a038e0bf2fc4725b/deferred/funca_oz8ghw DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-e7e7bd96e66759e6a038e0bf2fc4725b/deferred/funca_oz8ghw DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testSimultaneousReadsUncachedStream-gsn2xvwk/toilwf-e7e7bd96e66759e6a038e0bf2fc4725b DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-e7e7bd96e66759e6a038e0bf2fc4725b DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.02961444854736328 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-osgtn4if) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testSimultaneousReadsUncachedStream ___________ CachingFileStoreTestWithFileJobStore.testToilIsNotBroken ___________ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 120, in testToilIsNotBroken Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_uselessFunc' kind-_uselessFunc/instance-i79eq2kr v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testToilIsNotBroken (Apr 30 2024 11:59:49:273174 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testToilIsNotBroken ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testToilIsNotBroken-yp2xkzou/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testToilIsNotBroken-yp2xkzou/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-14js_bqi', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testToilIsNotBroken-yp2xkzou', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-14js_bqi'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'd60ddea0-47a6-45fb-8e46-ebae1eaf20e3' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testToilIsNotBroken-yp2xkzou/toilwf-738f31a9a6565b3a84817ffd45112d6d DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 2993339376. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 2993339376 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-36373bb6995c48dc8b1f4009a16f2572/stream for result of '_uselessFunc' TemporaryID(ede273d0-5ebf-44f9-b56b-0ec7b8b87e22) v0 DEBUG toil.job:job.py:2624 Saving graph of 4 jobs, 4 non-service, 4 new DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-uez5i9ho v0 DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-qmdtl2jh v0 DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-ni11s1gj v0 DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-i79eq2kr v0 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-uez5i9ho v1 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-qmdtl2jh v1 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-ni11s1gj v1 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-i79eq2kr v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'd60ddea0-47a6-45fb-8e46-ebae1eaf20e3', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-14js_bqi', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testToilIsNotBroken-yp2xkzou', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_uselessFunc/instance-i79eq2kr, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_uselessFunc/instance-i79eq2kr', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_uselessFunc' kind-_uselessFunc/instance-i79eq2kr v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9943640232086182 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5022299289703369 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 2993339376... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 2993339376 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-738f31a9a6565b3a84817ffd45112d6d/deferred/funckk1tlv84 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-738f31a9a6565b3a84817ffd45112d6d/deferred/funckk1tlv84 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testToilIsNotBroken-yp2xkzou/toilwf-738f31a9a6565b3a84817ffd45112d6d DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-738f31a9a6565b3a84817ffd45112d6d DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.026339292526245117 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-14js_bqi) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testToilIsNotBroken ____ CachingFileStoreTestWithFileJobStore.testWriteExportFileCompatibility _____ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 300, in testWriteExportFileCompatibility initialPermissions, fileID = toil.start(export_file_job) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-5xsequ7d v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testWriteExportFileCompatibility (Apr 30 2024 11:59:52:784042 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteExportFileCompatibility ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-yzy6r925/logFile'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-yzy6r925/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-d44w2nbc', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-yzy6r925', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-d44w2nbc'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'cf6b50bc-fdcb-44ea-8f29-b9adf9ad2eed' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-yzy6r925/toilwf-f7f69fdedbe152548a84076aa59a1571 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3025994320. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3025994320 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-1dae961a2dbb486c917cb0b7a13e917a/stream for result of '_testWriteExportFileCompatibility' TemporaryID(9e208ef4-df80-4075-b5f5-4ad5feb36a64) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-5xsequ7d v0 DEBUG toil.job:job.py:1299 New job version: '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-5xsequ7d v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'cf6b50bc-fdcb-44ea-8f29-b9adf9ad2eed', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-d44w2nbc', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-yzy6r925', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_testWriteExportFileCompatibility/instance-5xsequ7d, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_testWriteExportFileCompatibility/instance-5xsequ7d', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-5xsequ7d v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.992631435394287 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.0042917728424072266 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3025994320... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3025994320 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-f7f69fdedbe152548a84076aa59a1571/deferred/funcumt7ezko DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-f7f69fdedbe152548a84076aa59a1571/deferred/funcumt7ezko DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-yzy6r925/toilwf-f7f69fdedbe152548a84076aa59a1571 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-f7f69fdedbe152548a84076aa59a1571 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.0613400936126709 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-d44w2nbc) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteExportFileCompatibility _____________________________ JobTest.testStatic2 ______________________________ [gw2] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobTest.py", line 115, in testStatic2 Job.Runner.startToil(A, options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'fn1Test' kind-fn1Test/instance-i7yhpkis v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.jobTest.JobTest:testStatic2 (Apr 30 2024 11:59:53:603707 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.jobTest.JobTest.testStatic2 ... DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.jobTest.JobTest.testStatic2 ______ CachingFileStoreTestWithFileJobStore.testWriteLocalFileToJobStore _______ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 719, in testWriteLocalFileToJobStore Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-simz42wl v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testWriteLocalFileToJobStore (Apr 30 2024 11:59:55:760845 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteLocalFileToJobStore ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteLocalFileToJobStore-x7ebdmgf/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteLocalFileToJobStore-x7ebdmgf/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-2_ev7d4j', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteLocalFileToJobStore-x7ebdmgf', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-2_ev7d4j'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '3b0c69c4-8f5d-4da2-a5ea-2c0b630c00bd' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteLocalFileToJobStore-x7ebdmgf/toilwf-2fe39b6bcc2c545f81b04c9cfbfe0e3f DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3022657072. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3022657072 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-d45234d7634345fa90c3a0d0e1d0439d/stream for result of '_writeFileToJobStoreWithAsserts' TemporaryID(89bac773-beb6-46cc-8cf0-c5e076c4d166) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-simz42wl v0 DEBUG toil.job:job.py:1299 New job version: '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-simz42wl v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '3b0c69c4-8f5d-4da2-a5ea-2c0b630c00bd', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-2_ev7d4j', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteLocalFileToJobStore-x7ebdmgf', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_writeFileToJobStoreWithAsserts/instance-simz42wl, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_writeFileToJobStoreWithAsserts/instance-simz42wl', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-simz42wl v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9936103820800781 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.002306699752807617 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3022657072... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3022657072 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-2fe39b6bcc2c545f81b04c9cfbfe0e3f/deferred/funci628ygfa DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-2fe39b6bcc2c545f81b04c9cfbfe0e3f/deferred/funci628ygfa DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteLocalFileToJobStore-x7ebdmgf/toilwf-2fe39b6bcc2c545f81b04c9cfbfe0e3f DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-2fe39b6bcc2c545f81b04c9cfbfe0e3f DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.05623936653137207 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-2_ev7d4j) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteLocalFileToJobStore ___ NonCachingFileStoreTestWithFileJobStore.testReadWriteFileStreamTextMode ____ [gw2] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 351, in testReadWriteFileStreamTextMode toil.start(A) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-7yhq4myl v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore:testReadWriteFileStreamTextMode (Apr 30 2024 11:59:56:218447 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testReadWriteFileStreamTextMode ... DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'websocket', 'requests', 'boto3', 'urllib3', 'botocore', 'boto', 'docker', 'asyncio', 'concurrent', 'charset_normalizer', 'bcdocs', 'dill', 'debugWorkflow'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-dt_dltig/logFile'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-dt_dltig/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-9ydn_6iw', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-dt_dltig', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-9ydn_6iw'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '6c9e2482-9136-4a38-864d-4825e92a282b' DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-dt_dltig/toilwf-fe8a3c76857959d7b9f4caa18d550edb DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3033517552. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3033517552 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-b9589a35c26345d39a61fa6a43acf501/stream for result of '_testReadWriteFileStreamTextMode' TemporaryID(4e972c9a-1ca6-477b-acc0-9be3b9111c09) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-7yhq4myl v0 DEBUG toil.job:job.py:1299 New job version: '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-7yhq4myl v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '6c9e2482-9136-4a38-864d-4825e92a282b', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-9ydn_6iw', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-dt_dltig', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_testReadWriteFileStreamTextMode/instance-7yhq4myl, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_testReadWriteFileStreamTextMode/instance-7yhq4myl', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_testReadWriteFileStreamTextMode' kind-_testReadWriteFileStreamTextMode/instance-7yhq4myl v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9973063468933105 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.0024564266204833984 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3033517552... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3033517552 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-fe8a3c76857959d7b9f4caa18d550edb/deferred/func0ad7_tn6 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-fe8a3c76857959d7b9f4caa18d550edb/deferred/func0ad7_tn6 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testReadWriteFileStreamTextMode-dt_dltig/toilwf-fe8a3c76857959d7b9f4caa18d550edb DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-fe8a3c76857959d7b9f4caa18d550edb DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.015641212463378906 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-9ydn_6iw) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testReadWriteFileStreamTextMode _________ NonCachingFileStoreTestWithFileJobStore.testToilIsNotBroken __________ [gw2] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 120, in testToilIsNotBroken Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_uselessFunc' kind-_uselessFunc/instance-q46o3wy7 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore:testToilIsNotBroken (Apr 30 2024 11:59:59:078629 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testToilIsNotBroken ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'websocket', 'requests', 'boto3', 'urllib3', 'botocore', 'boto', 'docker', 'asyncio', 'concurrent', 'charset_normalizer', 'bcdocs', 'dill', 'debugWorkflow'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testToilIsNotBroken-znktbgq9/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'websocket', 'requests', 'boto3', 'urllib3', 'botocore', 'boto', 'docker', 'asyncio', 'concurrent', 'charset_normalizer', 'bcdocs', 'dill', 'debugWorkflow'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testToilIsNotBroken-znktbgq9/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-o4iw43if', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testToilIsNotBroken-znktbgq9', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-o4iw43if'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'f2d5da38-aab7-4863-8d1e-2d9193942bce' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testToilIsNotBroken-znktbgq9/toilwf-fce3c8cc257f5dedbab207d1ea6b0140 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3012875184. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3012875184 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-17e5da4934364a6b97ed826724fbf795/stream for result of '_uselessFunc' TemporaryID(2f250902-765d-483d-bcb9-83e148a250ed) v0 DEBUG toil.job:job.py:2624 Saving graph of 4 jobs, 4 non-service, 4 new DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-hrsgbtht v0 DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-ke_m937j v0 DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-8s1cdztf v0 DEBUG toil.job:job.py:2644 Processing job '_uselessFunc' kind-_uselessFunc/instance-q46o3wy7 v0 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-hrsgbtht v1 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-ke_m937j v1 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-8s1cdztf v1 DEBUG toil.job:job.py:1299 New job version: '_uselessFunc' kind-_uselessFunc/instance-q46o3wy7 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'f2d5da38-aab7-4863-8d1e-2d9193942bce', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-o4iw43if', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testToilIsNotBroken-znktbgq9', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_uselessFunc/instance-q46o3wy7, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_uselessFunc/instance-q46o3wy7', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_uselessFunc' kind-_uselessFunc/instance-q46o3wy7 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9976086616516113 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.002812623977661133 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3012875184... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3012875184 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-fce3c8cc257f5dedbab207d1ea6b0140/deferred/func2qplxg70 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-fce3c8cc257f5dedbab207d1ea6b0140/deferred/func2qplxg70 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testToilIsNotBroken-znktbgq9/toilwf-fce3c8cc257f5dedbab207d1ea6b0140 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-fce3c8cc257f5dedbab207d1ea6b0140 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.02188897132873535 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-o4iw43if) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testToilIsNotBroken _____ CachingFileStoreTestWithFileJobStore.testWriteNonLocalFileToJobStore _____ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 711, in testWriteNonLocalFileToJobStore Job.Runner.startToil(A, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-co4zvebl v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testWriteNonLocalFileToJobStore (Apr 30 2024 11:59:58:721056 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteNonLocalFileToJobStore ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteNonLocalFileToJobStore-vihh_q4r/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteNonLocalFileToJobStore-vihh_q4r/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-pvbfc7y8', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteNonLocalFileToJobStore-vihh_q4r', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-pvbfc7y8'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '8d439743-eb28-4092-b844-641a522a107a' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteNonLocalFileToJobStore-vihh_q4r/toilwf-cc10cefaad3c5663b30147b3ca50e3cc DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 2993462160. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 2993462160 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-1bf9333420984b5099bb970703bac245/stream for result of '_writeFileToJobStoreWithAsserts' TemporaryID(ce5abbd2-f675-4c00-aab0-5c0a2777be15) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-co4zvebl v0 DEBUG toil.job:job.py:1299 New job version: '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-co4zvebl v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '8d439743-eb28-4092-b844-641a522a107a', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-pvbfc7y8', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteNonLocalFileToJobStore-vihh_q4r', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_writeFileToJobStoreWithAsserts/instance-co4zvebl, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_writeFileToJobStoreWithAsserts/instance-co4zvebl', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_writeFileToJobStoreWithAsserts' kind-_writeFileToJobStoreWithAsserts/instance-co4zvebl v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9945704936981201 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5031888484954834 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 2993462160... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 2993462160 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-cc10cefaad3c5663b30147b3ca50e3cc/deferred/func_judpyr9 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-cc10cefaad3c5663b30147b3ca50e3cc/deferred/func_judpyr9 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteNonLocalFileToJobStore-vihh_q4r/toilwf-cc10cefaad3c5663b30147b3ca50e3cc DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-cc10cefaad3c5663b30147b3ca50e3cc DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.03255128860473633 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-pvbfc7y8) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteNonLocalFileToJobStore ___ NonCachingFileStoreTestWithFileJobStore.testWriteExportFileCompatibility ___ [gw2] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 300, in testWriteExportFileCompatibility initialPermissions, fileID = toil.start(export_file_job) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-b9h531g6 v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore:testWriteExportFileCompatibility (Apr 30 2024 12:00:01:995937 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testWriteExportFileCompatibility ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'websocket', 'requests', 'boto3', 'urllib3', 'botocore', 'boto', 'docker', 'asyncio', 'concurrent', 'charset_normalizer', 'bcdocs', 'dill', 'debugWorkflow'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-0ayvg0o_/logFile'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-0ayvg0o_/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-ju1eid1m', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-0ayvg0o_', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-ju1eid1m'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '79bc5769-66e5-40b2-8fa8-95f06e5b6fce' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-0ayvg0o_/toilwf-20508f7f18c05be6b471a76db45684cc DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3021921840. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3021921840 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-e8bf5795c2a5447195f1bd5a719bc4be/stream for result of '_testWriteExportFileCompatibility' TemporaryID(f270151c-1d76-4e2d-a57d-901009abea98) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-b9h531g6 v0 DEBUG toil.job:job.py:1299 New job version: '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-b9h531g6 v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '79bc5769-66e5-40b2-8fa8-95f06e5b6fce', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-ju1eid1m', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-0ayvg0o_', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_testWriteExportFileCompatibility/instance-b9h531g6, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_testWriteExportFileCompatibility/instance-b9h531g6', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_testWriteExportFileCompatibility' kind-_testWriteExportFileCompatibility/instance-b9h531g6 v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9967796802520752 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.0028100013732910156 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3021921840... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3021921840 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-20508f7f18c05be6b471a76db45684cc/deferred/funcs3yjtraa DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-20508f7f18c05be6b471a76db45684cc/deferred/funcs3yjtraa DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteExportFileCompatibility-0ayvg0o_/toilwf-20508f7f18c05be6b471a76db45684cc DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-20508f7f18c05be6b471a76db45684cc DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.03118300437927246 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-ju1eid1m) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testWriteExportFileCompatibility ___ CachingFileStoreTestWithFileJobStore.testWriteReadGlobalFilePermissions ____ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 269, in testWriteReadGlobalFilePermissions Job.Runner.startToil(read_write_job, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-ocpanqrt v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore:testWriteReadGlobalFilePermissions (Apr 30 2024 12:00:02:140577 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteReadGlobalFilePermissions ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-8zh2uxpm/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-8zh2uxpm/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-f7mtz74e', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-8zh2uxpm', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-f7mtz74e'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '3a4fc93b-1f51-4a0b-b6a9-e46ec6df0c99' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-8zh2uxpm/toilwf-6b5e8066676a5f00aadde8bd4a961a35 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3033284144. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3033284144 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-b75f0da1bca4441bb3ace08f25811fc4/stream for result of '_testWriteReadGlobalFilePermissions' TemporaryID(78852bfa-f969-4c40-90fb-39f3ffadd710) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-ocpanqrt v0 DEBUG toil.job:job.py:1299 New job version: '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-ocpanqrt v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '3a4fc93b-1f51-4a0b-b6a9-e46ec6df0c99', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-f7mtz74e', 'workDir': '/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-8zh2uxpm', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_testWriteReadGlobalFilePermissions/instance-ocpanqrt, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_testWriteReadGlobalFilePermissions/instance-ocpanqrt', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-ocpanqrt v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9943158626556396 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5022566318511963 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3033284144... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3033284144 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-6b5e8066676a5f00aadde8bd4a961a35/deferred/funcc55cyy0f DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-6b5e8066676a5f00aadde8bd4a961a35/deferred/funcc55cyy0f DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-8zh2uxpm/toilwf-6b5e8066676a5f00aadde8bd4a961a35 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-6b5e8066676a5f00aadde8bd4a961a35 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.0570530891418457 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.CachingFileStoreTestWithFileJobStore-jobstore-f7mtz74e) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore.testWriteReadGlobalFilePermissions __ NonCachingFileStoreTestWithFileJobStore.testWriteReadGlobalFilePermissions __ [gw2] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/fileStoreTest.py", line 269, in testWriteReadGlobalFilePermissions Job.Runner.startToil(read_write_job, self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-odravbor v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore:testWriteReadGlobalFilePermissions (Apr 30 2024 12:00:04:898542 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testWriteReadGlobalFilePermissions ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'websocket', 'requests', 'boto3', 'urllib3', 'botocore', 'boto', 'docker', 'asyncio', 'concurrent', 'charset_normalizer', 'bcdocs', 'dill', 'debugWorkflow'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-z8upoa7_/logFile'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'websocket', 'requests', 'boto3', 'urllib3', 'botocore', 'boto', 'docker', 'asyncio', 'concurrent', 'charset_normalizer', 'bcdocs', 'dill', 'debugWorkflow'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-z8upoa7_/logFile', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-93kmczat', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-z8upoa7_', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-93kmczat'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '8c50e2d5-4bee-480a-aaa2-a9b0339fedfc' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-z8upoa7_/toilwf-c0e23b8bbce859d885091a778f28b655 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3027899504. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3027899504 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.fileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-26f183886d494f25a627d7565a174c40/stream for result of '_testWriteReadGlobalFilePermissions' TemporaryID(8ba73469-9158-4da5-8e43-2f775da51319) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-odravbor v0 DEBUG toil.job:job.py:1299 New job version: '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-odravbor v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '8c50e2d5-4bee-480a-aaa2-a9b0339fedfc', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-93kmczat', 'workDir': '/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-z8upoa7_', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} INFO toil.realtimeLogger:realtimeLogger.py:139 Starting real-time logging. DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-_testWriteReadGlobalFilePermissions/instance-odravbor, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-_testWriteReadGlobalFilePermissions/instance-odravbor', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job '_testWriteReadGlobalFilePermissions' kind-_testWriteReadGlobalFilePermissions/instance-odravbor v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9964792728424072 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.002279520034790039 seconds INFO toil.realtimeLogger:realtimeLogger.py:176 Stopping real-time logging server. INFO toil.realtimeLogger:realtimeLogger.py:180 Joining real-time logging server thread. DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3027899504... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3027899504 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-c0e23b8bbce859d885091a778f28b655/deferred/funcqx3pwod4 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-c0e23b8bbce859d885091a778f28b655/deferred/funcqx3pwod4 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-testWriteReadGlobalFilePermissions-z8upoa7_/toilwf-c0e23b8bbce859d885091a778f28b655 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-c0e23b8bbce859d885091a778f28b655 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.023186206817626953 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore-jobstore-93kmczat) INFO toil.test:__init__.py:120 Tore down toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore.testWriteReadGlobalFilePermissions ____________________ JobFileStoreTest.testCachingFileStore _____________________ [gw2] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobFileStoreTest.py", line 38, in testCachingFileStore workflow.start(Job.wrapJobFn(simpleFileStoreJob)) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'simpleFileStoreJob' kind-simpleFileStoreJob/instance-o67nomow v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.jobFileStoreTest.JobFileStoreTest:testCachingFileStore (Apr 30 2024 12:00:14:922818 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.jobFileStoreTest.JobFileStoreTest.testCachingFileStore ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.jobFileStoreTest.JobFileStoreTest.testCachingFileStore _____________________ UtilsTest.testGetStatusFailedToilWF ______________________ [gw3] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/utilsTest.py", line 336, in testGetStatusFailedToilWF self.check_status('ERROR', status_fn=ToilStatus.getStatus) File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/utilsTest.py", line 312, in check_status self.assertEqual(s, status, f'Waited {seconds} seconds without status reaching {status}; stuck at {s}') File "/usr/lib/python3.11/unittest/case.py", line 873, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib/python3.11/unittest/case.py", line 1253, in assertMultiLineEqual self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib/python3.11/unittest/case.py", line 703, in fail raise self.failureException(msg) AssertionError: 'RUNNING' != 'ERROR' - RUNNING + ERROR : Waited 20 seconds without status reaching ERROR; stuck at RUNNING ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.utils.utilsTest.UtilsTest:testGetStatusFailedToilWF (Apr 30 2024 11:59:48:200158 PDT) ----------------------------- Captured stdout call ----------------------------- No sort file specified. Generating one automatically called: fileToSort.txt. ----------------------------- Captured stderr call ----------------------------- [2024-04-30T11:59:52-1200] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. [2024-04-30T11:59:52-1200] [MainThread] [I] [toil.leader] Issued job 'setup' kind-setup/instance-m6cuy7vo v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 572.2 Mi, cores: 1, accelerators: [], preemptible: False [2024-04-30T11:59:54-1200] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run User defined signal 1 [2024-04-30T11:59:54-1200] [Thread-1 (daddy)] [E] [toil.batchSystems.singleMachine] Got exit code 138 (indicating failure) from job _toil_worker setup file:/tmp/toil-test-utils.utilsTest.UtilsTest-testGetStatusFailedToilWF-u9cjsolu/jobstore kind-setup/instance-m6cuy7vo. [2024-04-30T11:59:54-1200] [MainThread] [W] [toil.leader] Job failed with exit value 138: 'setup' kind-setup/instance-m6cuy7vo v1 Exit reason: None [2024-04-30T11:59:54-1200] [MainThread] [W] [toil.leader] No log file is present, despite job failing: 'setup' kind-setup/instance-m6cuy7vo v1 [2024-04-30T11:59:54-1200] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'setup' kind-setup/instance-m6cuy7vo v1 with ID kind-setup/instance-m6cuy7vo to 1 [2024-04-30T11:59:54-1200] [MainThread] [W] [toil.job] We have increased the default memory of the failed job 'setup' kind-setup/instance-m6cuy7vo v1 to 2147483648 bytes Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/build/reproducible-path/toil-6.1.0/src/toil/test/sort/sort.py", line 260, in main() File "/build/reproducible-path/toil-6.1.0/src/toil/test/sort/sort.py", line 248, in main sortedFileID = workflow.start(Job.wrapJobFn(setup, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'setup' kind-setup/instance-m6cuy7vo v2 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. __________________________ UtilsTest.testPrintJobLog ___________________________ [gw3] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/usr/lib/python3.11/unittest/mock.py", line 1378, in patched return func(*newargs, **newkeywargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/utilsTest.py", line 376, in testPrintJobLog self.assertIn('invalidcommand', args[0]) File "/usr/lib/python3.11/unittest/case.py", line 1140, in assertIn self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib/python3.11/unittest/case.py", line 703, in fail raise self.failureException(msg) AssertionError: 'invalidcommand' not found in "LOG_FILE_OF_JOB: 'CWLJob' alwaysfails.cwl kind-CWLJob/instance-ui3ou82_ v1 LOG: Job has no log file" ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.utils.utilsTest.UtilsTest:testPrintJobLog (Apr 30 2024 12:00:17:606784 PDT) ----------------------------- Captured stderr call ----------------------------- WARNING: You are using cwltool version 3.1.20240404144621, which is not the version Toil is tested against. To install the correct cwltool for Toil, do: pip install cwltool==3.1.20240112164112 [2024-04-30T12:00:23-1200] [MainThread] [I] [cwltool] Resolved 'src/toil/test/cwl/alwaysfails.cwl' to 'file:///build/reproducible-path/toil-6.1.0/src/toil/test/cwl/alwaysfails.cwl' [2024-04-30T12:00:29-1200] [MainThread] [I] [toil.cwl.cwltoil] Importing input files... [2024-04-30T12:00:29-1200] [MainThread] [I] [toil.cwl.cwltoil] Importing tool-associated files... [2024-04-30T12:00:29-1200] [MainThread] [I] [toil.cwl.cwltoil] Creating root job [2024-04-30T12:00:29-1200] [MainThread] [I] [toil.cwl.cwltoil] Starting workflow [2024-04-30T12:00:29-1200] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. Traceback (most recent call last): File "/build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/.local/bin/toil-cwl-runner", line 33, in sys.exit(load_entry_point('toil', 'console_scripts', 'toil-cwl-runner')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/cwl/cwltoil.py", line 3870, in main outobj = toil.start(wf1) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'CWLJob' alwaysfails.cwl kind-CWLJob/instance-ui3ou82_ v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. __________________ UserDefinedJobArgTypeTest.testJobFunction ___________________ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/userDefinedJobArgTypeTest.py", line 39, in testJobFunction Job.Runner.startToil(Job.wrapJobFn(jobFunction, 0, Foo()), self.options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'jobFunction' kind-jobFunction/instance-3sb_wbwt v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.userDefinedJobArgTypeTest.UserDefinedJobArgTypeTest:testJobFunction (Apr 30 2024 12:00:35:149695 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.userDefinedJobArgTypeTest.UserDefinedJobArgTypeTest.testJobFunction ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.userDefinedJobArgTypeTest.UserDefinedJobArgTypeTest.testJobFunction ______________________ JobFileStoreTest.testJobFileStore _______________________ [gw2] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobFileStoreTest.py", line 77, in testJobFileStore self._testJobFileStore(retryCount=0, badWorker=0.0, stringNo=5, stringLength=1000000) File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobFileStoreTest.py", line 69, in _testJobFileStore Job.Runner.startToil(Job.wrapJobFn(fileTestJob, [], File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'fileTestJob' kind-fileTestJob/instance-hwgrajwz v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.jobFileStoreTest.JobFileStoreTest:testJobFileStore (Apr 30 2024 12:00:17:314081 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.jobFileStoreTest.JobFileStoreTest.testJobFileStore ... DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'websocket', 'requests', 'boto3', 'urllib3', 'botocore', 'boto', 'docker', 'asyncio', 'concurrent', 'charset_normalizer', 'bcdocs', 'dill', 'debugWorkflow'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'websocket', 'requests', 'boto3', 'urllib3', 'botocore', 'boto', 'docker', 'asyncio', 'concurrent', 'charset_normalizer', 'bcdocs', 'dill', 'debugWorkflow'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': None, 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.jobFileStoreTest.JobFileStoreTest-jobstore-az0ct5ca', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 1.0, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.jobFileStoreTest.JobFileStoreTest-jobstore-az0ct5ca'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: 'ca4ea265-0354-4a70-b4bc-8ad3e34f3b4d' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toilwf-1fb42347de6952f39a079e084ae0fc34 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 2992608816. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 2992608816 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.jobFileStoreTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-610e10b1e9824c18a170e9987df8e9cd/stream for result of 'fileTestJob' TemporaryID(33574a21-8651-4c6d-98a5-342740b597ec) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job 'fileTestJob' kind-fileTestJob/instance-hwgrajwz v0 DEBUG toil.job:job.py:1299 New job version: 'fileTestJob' kind-fileTestJob/instance-hwgrajwz v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': 'ca4ea265-0354-4a70-b4bc-8ad3e34f3b4d', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.jobFileStoreTest.JobFileStoreTest-jobstore-az0ct5ca', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'onSuccess', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 1.0, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} DEBUG toil.realtimeLogger:realtimeLogger.py:162 Real-time logging disabled DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-fileTestJob/instance-hwgrajwz, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-fileTestJob/instance-hwgrajwz', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job 'fileTestJob' kind-fileTestJob/instance-hwgrajwz v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9958600997924805 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5011215209960938 seconds DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 2992608816... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 2992608816 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-1fb42347de6952f39a079e084ae0fc34/deferred/funcoa9i8knj DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-1fb42347de6952f39a079e084ae0fc34/deferred/funcoa9i8knj DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toilwf-1fb42347de6952f39a079e084ae0fc34 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-1fb42347de6952f39a079e084ae0fc34 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.02270364761352539 seconds. INFO toil.test:__init__.py:120 Tore down toil.test.src.jobFileStoreTest.JobFileStoreTest.testJobFileStore _________________ MessageBusTest.test_restart_without_bus_path _________________ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/busTest.py", line 127, in test_restart_without_bus_path toil.start(root) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'failing_job_fn' kind-failing_job_fn/instance-k5in4fku v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.busTest.MessageBusTest:test_restart_without_bus_path (Apr 30 2024 12:00:41:565887 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.busTest.MessageBusTest.test_restart_without_bus_path ... DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': None, 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.busTest.MessageBusTest-jobstore-6edqlvdb', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'never', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': '/tmp/toil-test-src.busTest.MessageBusTest-test_restart_without_bus_path-tempDir-rgpk09wc/bus_holder/messagebus.txt', 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.busTest.MessageBusTest-jobstore-6edqlvdb'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '4fd0113f-aad6-4e24-814e-823a9c45eb3d' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toilwf-e1a97ed1deb55b6c94417e645ca09be2 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3022554704. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3022554704 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.busTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-bc620e0a1a434b5489eb45137756b7a2/stream for result of 'failing_job_fn' TemporaryID(bef86c36-12b7-422b-8143-ad34540bcaf4) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job 'failing_job_fn' kind-failing_job_fn/instance-k5in4fku v0 DEBUG toil.job:job.py:1299 New job version: 'failing_job_fn' kind-failing_job_fn/instance-k5in4fku v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '4fd0113f-aad6-4e24-814e-823a9c45eb3d', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.busTest.MessageBusTest-jobstore-6edqlvdb', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'never', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': '/tmp/toil-test-src.busTest.MessageBusTest-test_restart_without_bus_path-tempDir-rgpk09wc/bus_holder/messagebus.txt', 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} DEBUG toil.realtimeLogger:realtimeLogger.py:162 Real-time logging disabled DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-failing_job_fn/instance-k5in4fku, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-failing_job_fn/instance-k5in4fku', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job 'failing_job_fn' kind-failing_job_fn/instance-k5in4fku v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9934968948364258 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.49919962882995605 seconds DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3022554704... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3022554704 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-e1a97ed1deb55b6c94417e645ca09be2/deferred/funcp7v7hc_u DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-e1a97ed1deb55b6c94417e645ca09be2/deferred/funcp7v7hc_u DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toilwf-e1a97ed1deb55b6c94417e645ca09be2 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-e1a97ed1deb55b6c94417e645ca09be2 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.034447669982910156 seconds. INFO toil.test:__init__.py:120 Tore down toil.test.src.busTest.MessageBusTest.test_restart_without_bus_path ___________________ JobFileStoreTest.testNonCachingFileStore ___________________ [gw2] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobFileStoreTest.py", line 44, in testNonCachingFileStore workflow.start(Job.wrapJobFn(simpleFileStoreJob)) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'simpleFileStoreJob' kind-simpleFileStoreJob/instance-roip_c_o v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.jobFileStoreTest.JobFileStoreTest:testNonCachingFileStore (Apr 30 2024 12:00:42:741516 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.jobFileStoreTest.JobFileStoreTest.testNonCachingFileStore ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.jobFileStoreTest.JobFileStoreTest.testNonCachingFileStore ___________________ CheckpointTest.testCheckpointNotRetried ____________________ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/checkpointTest.py", line 33, in testCheckpointNotRetried Job.Runner.startToil(CheckRetryCount(numFailuresBeforeSuccess=1), options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'CheckRetryCount' kind-CheckRetryCount/instance-ezuqf7ne v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.checkpointTest.CheckpointTest:testCheckpointNotRetried (Apr 30 2024 12:00:44:553682 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.checkpointTest.CheckpointTest.testCheckpointNotRetried ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.checkpointTest.CheckpointTest.testCheckpointNotRetried ____________________ JobEncapsulationTest.testEncapsulation ____________________ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/jobEncapsulationTest.py", line 43, in testEncapsulation Job.Runner.startToil(a, options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'EncapsulatedJob' a-encap kind-EncapsulatedJob/instance-lwops2jb v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.jobEncapsulationTest.JobEncapsulationTest:testEncapsulation (Apr 30 2024 12:00:50:352154 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.jobEncapsulationTest.JobEncapsulationTest.testEncapsulation ... INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.jobEncapsulationTest.JobEncapsulationTest.testEncapsulation ________________________ UtilsTest.testRestartAttribute ________________________ [gw3] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/utilsTest.py", line 399, in testRestartAttribute self.assertFalse(os.path.exists(self.toilDir)) File "/usr/lib/python3.11/unittest/case.py", line 709, in assertFalse raise self.failureException(msg) AssertionError: True is not false ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.utils.utilsTest.UtilsTest:testRestartAttribute (Apr 30 2024 12:00:33:162792 PDT) ----------------------------- Captured stdout call ----------------------------- No sort file specified. Generating one automatically called: fileToSort.txt. Sorting existing file: fileToSort.txt Sorting existing file: fileToSort.txt ----------------------------- Captured stderr call ----------------------------- [2024-04-30T12:00:37-1200] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. [2024-04-30T12:00:37-1200] [MainThread] [I] [toil.leader] Issued job 'setup' kind-setup/instance-28wvc7zm v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 572.2 Mi, cores: 1, accelerators: [], preemptible: False [2024-04-30T12:00:39-1200] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run User defined signal 1 [2024-04-30T12:00:40-1200] [Thread-1 (daddy)] [E] [toil.batchSystems.singleMachine] Got exit code 138 (indicating failure) from job _toil_worker setup file:/tmp/toil-test-utils.utilsTest.UtilsTest-testRestartAttribute-pv8_123s/jobstore kind-setup/instance-28wvc7zm. [2024-04-30T12:00:40-1200] [MainThread] [W] [toil.leader] Job failed with exit value 138: 'setup' kind-setup/instance-28wvc7zm v1 Exit reason: None [2024-04-30T12:00:40-1200] [MainThread] [W] [toil.leader] No log file is present, despite job failing: 'setup' kind-setup/instance-28wvc7zm v1 [2024-04-30T12:00:40-1200] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'setup' kind-setup/instance-28wvc7zm v1 with ID kind-setup/instance-28wvc7zm to 1 [2024-04-30T12:00:40-1200] [MainThread] [W] [toil.job] We have increased the default memory of the failed job 'setup' kind-setup/instance-28wvc7zm v1 to 2147483648 bytes Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/build/reproducible-path/toil-6.1.0/src/toil/test/sort/restart_sort.py", line 264, in main() File "/build/reproducible-path/toil-6.1.0/src/toil/test/sort/restart_sort.py", line 253, in main sortedFileID = workflow.start(Job.wrapJobFn(setup, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'setup' kind-setup/instance-28wvc7zm v2 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. [2024-04-30T12:00:50-1200] [MainThread] [C] [toil.jobStores.abstractJobStore] Repairing job: kind-setup/instance-28wvc7zm [2024-04-30T12:00:50-1200] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/build/reproducible-path/toil-6.1.0/src/toil/test/sort/sort.py", line 260, in main() File "/build/reproducible-path/toil-6.1.0/src/toil/test/sort/sort.py", line 255, in main sortedFileID = workflow.restart() ^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 952, in restart return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'setup' kind-setup/instance-28wvc7zm v3 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. ________________________ ResumabilityTest.test_chaining ________________________ [gw0] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/resumabilityTest.py", line 71, in test_chaining Job.Runner.startToil(root, options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'chaining_parent' kind-chaining_parent/instance-ev5ik0ht v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.resumabilityTest.ResumabilityTest:test_chaining (Apr 30 2024 12:00:52:848324 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.resumabilityTest.ResumabilityTest.test_chaining ... DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:281 Logging to file '/tmp/toil-test-src.resumabilityTest.ResumabilityTest-test_chaining-eumecfcc/log.txt'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'concurrent', 'docker', 'bcdocs', 'debugWorkflow', 'websocket', 'boto', 'boto3', 'dill', 'botocore', 'requests', 'urllib3', 'charset_normalizer', 'asyncio'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'DEBUG', 'logFile': '/tmp/toil-test-src.resumabilityTest.ResumabilityTest-test_chaining-eumecfcc/log.txt', 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-src.resumabilityTest.ResumabilityTest-jobstore-9a2vz7vd', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-src.resumabilityTest.ResumabilityTest-jobstore-9a2vz7vd'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '1f0e8afa-677a-41ab-aabb-2ef8b552c612' DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toilwf-79486330409b592b8f6a626892c531b3 DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3040878896. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3040878896 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.src.resumabilityTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-691b0dc9dba14ef8afbf3e1539268a16/stream for result of 'chaining_parent' TemporaryID(0c002757-a01d-4d64-88ac-00b752911e74) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job 'chaining_parent' kind-chaining_parent/instance-ev5ik0ht v0 DEBUG toil.job:job.py:1299 New job version: 'chaining_parent' kind-chaining_parent/instance-ev5ik0ht v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '1f0e8afa-677a-41ab-aabb-2ef8b552c612', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-src.resumabilityTest.ResumabilityTest-jobstore-9a2vz7vd', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'onSuccess', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'DEBUG', 'workflowAttemptNumber': 0} DEBUG toil.realtimeLogger:realtimeLogger.py:162 Real-time logging disabled DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-chaining_parent/instance-ev5ik0ht, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-chaining_parent/instance-ev5ik0ht', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job 'chaining_parent' kind-chaining_parent/instance-ev5ik0ht v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9910383224487305 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.5009539127349854 seconds DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3040878896... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3040878896 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-79486330409b592b8f6a626892c531b3/deferred/funca28838fb DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-79486330409b592b8f6a626892c531b3/deferred/funca28838fb DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toilwf-79486330409b592b8f6a626892c531b3 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-79486330409b592b8f6a626892c531b3 DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.04462242126464844 seconds. INFO toil.test:__init__.py:120 Tore down toil.test.src.resumabilityTest.ResumabilityTest.test_chaining _________________________ UtilsTest.testUnicodeSupport _________________________ [gw3] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/utilsTest.py", line 287, in testUnicodeSupport Job.Runner.startToil(Job.wrapFn(printUnicodeCharacter), options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'printUnicodeCharacter' kind-printUnicodeCharacter/instance-1d6was5m v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.utils.utilsTest.UtilsTest:testUnicodeSupport (Apr 30 2024 12:00:53:468052 PDT) ------------------------------ Captured log call ------------------------------- DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'bcdocs', 'asyncio', 'requests', 'urllib3', 'boto3', 'charset_normalizer', 'botocore', 'dill', 'websocket', 'boto', 'concurrent', 'docker', 'debugWorkflow'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.statsAndLogging:statsAndLogging.py:328 Suppressing the following loggers: {'bcdocs', 'asyncio', 'requests', 'urllib3', 'boto3', 'charset_normalizer', 'botocore', 'dill', 'websocket', 'boto', 'concurrent', 'docker', 'debugWorkflow'} DEBUG toil.statsAndLogging:statsAndLogging.py:295 Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'. DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'INFO', 'logFile': None, 'logRotating': False, 'jobStore': None, 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.common:common.py:414 Loaded configuration: {'config_version': '1.0', 'config': None, 'logLevel': 'debug', 'logFile': None, 'logRotating': False, 'jobStore': 'file:/tmp/toil-test-utils.utilsTest.UtilsTest-jobstore-lgaxlma7', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'coalesceStatusCalls': True, 'statePollingWait': None, 'batch_logs_dir': None, 'scale': 1, 'allocate_mem': True, 'symlinkImports': True, 'moveOutputs': False, 'caching': None, 'provisioner': None, 'nodeTypes': [], 'maxNodes': [10], 'minNodes': [0], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'defaultPreemptible': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'realTimeLogging': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'environment': {}, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'linkImports': None, 'moveExports': None, 'enableCaching': None, 'not_strict': False, 'enable_dev': False, 'enable_ext': False, 'quiet': False, 'basedir': None, 'outdir': None, 'log_dir': '', 'user_space_docker_cmd': None, 'singularity': False, 'podman': False, 'no_container': False, 'rm_container': True, 'custom_net': None, 'record_container_id': False, 'cidfile_dir': None, 'cidfile_prefix': None, 'preserve_environment': ('PATH',), 'preserve_entire_environment': False, 'destBucket': None, 'beta_dependency_resolvers_configuration': None, 'beta_dependencies_directory': None, 'beta_use_biocontainers': None, 'beta_conda_dependencies': None, 'tmpdir_prefix': None, 'tmp_outdir_prefix': None, 'force_docker_pull': False, 'no_match_user': False, 'no_read_only': False, 'strict_memory_limit': False, 'strict_cpu_limit': False, 'relax_path_checks': False, 'default_container': None, 'do_validate': True, 'fast_parser': False, 'compute_checksum': True, 'eval_timeout': 20, 'overrides': None, 'mpi_config_file': None, 'bypass_file_store': False, 'reference_inputs': False, 'disable_streaming': False, 'provenance': None, 'user_provenance': False, 'host_provenance': False, 'orcid': '', 'cwl_full_name': '', 'output_dialect': 'cromwell', 'output_directory': None, 'output_file': None, 'container': 'auto'} DEBUG toil.jobStores.fileJobStore:fileJobStore.py:88 Path to job store directory is '/tmp/toil-test-utils.utilsTest.UtilsTest-jobstore-lgaxlma7'. DEBUG toil.jobStores.abstractJobStore:abstractJobStore.py:204 The workflow ID is: '9e8d0155-f9cf-4703-81c9-6a8b9336b68b' DEBUG toil.common:common.py:740 Obtained node ID 9d1f8bcc-f705-4f67-84fb-c68d8c6c626b from file /proc/sys/kernel/random/boot_id DEBUG toil.common:common.py:1041 Using the single machine batch system DEBUG toil.common:common.py:1339 Created the workflow directory for this machine at /tmp/toilwf-d878f8b1706f583d9a0f739c1957d9ec DEBUG toil.batchSystems.singleMachine:singleMachine.py:217 Started daddy thread for batch system 3023285232. DEBUG toil.batchSystems.singleMachine:singleMachine.py:199 Started batch system 3023285232 in normal mode. DEBUG toil.common:common.py:1059 User script ModuleDescriptor(dirPath='/build/reproducible-path/toil-6.1.0/src', name='toil.test.utils.utilsTest', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it. DEBUG toil.common:common.py:1087 No user script to auto-deploy. DEBUG toil.common:common.py:1222 Written the environment for the jobs to the environment file DEBUG toil.common:common.py:1226 Caching all jobs in job store DEBUG toil.common:common.py:1228 0 jobs downloaded. DEBUG toil.job:job.py:1924 Issuing promise files/no-job/file-0b3b02f0890d4788a28da8b1bea6a588/stream for result of 'printUnicodeCharacter' TemporaryID(2de3bd4a-61ac-4f14-a176-be9a415fdcf1) v0 DEBUG toil.job:job.py:2624 Saving graph of 1 jobs, 1 non-service, 1 new DEBUG toil.job:job.py:2644 Processing job 'printUnicodeCharacter' kind-printUnicodeCharacter/instance-1d6was5m v0 DEBUG toil.job:job.py:1299 New job version: 'printUnicodeCharacter' kind-printUnicodeCharacter/instance-1d6was5m v1 INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. DEBUG toil:__init__.py:439 Configuration: {'cwl': False, 'workflowID': '9e8d0155-f9cf-4703-81c9-6a8b9336b68b', 'kill_polling_interval': 5, 'jobStore': 'file:/tmp/toil-test-utils.utilsTest.UtilsTest-jobstore-lgaxlma7', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 1, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'debug', 'workflowAttemptNumber': 0} DEBUG toil.realtimeLogger:realtimeLogger.py:162 Real-time logging disabled DEBUG toil.bus:bus.py:377 Listening for message topic: toil.bus.JobUpdatedMessage DEBUG toil.toilState:toilState.py:261 Found job to run: kind-printUnicodeCharacter/instance-1d6was5m, with command: True, with checkpoint: False, with services: False, with no next successors: False DEBUG toil.bus:bus.py:363 Notifying toil.bus.JobUpdatedMessage with message: JobUpdatedMessage(job_id='kind-printUnicodeCharacter/instance-1d6was5m', result_status=0) DEBUG toil.leader:leader.py:141 Found 1 jobs to start and 0 jobs with successors to run DEBUG toil.leader:leader.py:148 Checked batch system has no running jobs and no updated jobs DEBUG toil.serviceManager:serviceManager.py:34 Initializing service manager DEBUG toil.leader:leader.py:614 Built the jobs list, currently have 1 jobs to update and 0 jobs issued DEBUG toil.leader:leader.py:507 Updating status of job 'printUnicodeCharacter' kind-printUnicodeCharacter/instance-1d6was5m v1 with result status: 0 DEBUG toil.serviceManager:serviceManager.py:229 Waiting for service manager thread to finish ... DEBUG toil.serviceManager:serviceManager.py:254 Received signal to quit starting services. DEBUG toil.serviceManager:serviceManager.py:236 ... finished shutting down the service manager. Took 1.9978690147399902 seconds DEBUG toil.statsAndLogging:statsAndLogging.py:226 Waiting for stats and logging collator thread to finish ... DEBUG toil.statsAndLogging:statsAndLogging.py:230 ... finished collating stats and logs. Took 0.0038101673126220703 seconds DEBUG toil.common:common.py:1401 Shutting down batch system ... DEBUG toil.batchSystems.singleMachine:singleMachine.py:270 Daddy thread cleaning up 0 remaining children for batch system 3023285232... DEBUG toil.batchSystems.singleMachine:singleMachine.py:274 Daddy thread for batch system 3023285232 finishing because no children should now exist DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:474 Attempting worker cleanup DEBUG toil.deferred:deferred.py:203 Cleaning up deferred functions system DEBUG toil.deferred:deferred.py:145 Opened with own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-d878f8b1706f583d9a0f739c1957d9ec/deferred/funck60yybh7 DEBUG toil.deferred:deferred.py:274 Running orphaned deferred functions DEBUG toil.deferred:deferred.py:351 Ran orphaned deferred functions from 0 abandoned state files DEBUG toil.deferred:deferred.py:153 Removing own state file /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-d878f8b1706f583d9a0f739c1957d9ec/deferred/funck60yybh7 DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:484 Deleting workflow directory /tmp/toilwf-d878f8b1706f583d9a0f739c1957d9ec DEBUG toil.batchSystems.abstractBatchSystem:abstractBatchSystem.py:488 Deleting coordination directory /tmp/dh-xdg-rundir-jksNxF0D/toil/toilwf-d878f8b1706f583d9a0f739c1957d9ec DEBUG toil.common:common.py:1403 ... finished shutting down the batch system in 0.03525257110595703 seconds. INFO toil.common:common.py:860 Successfully deleted the job store: FileJobStore(/tmp/toil-test-utils.utilsTest.UtilsTest-jobstore-lgaxlma7) INFO toil.test:__init__.py:120 Tore down toil.test.utils.utilsTest.UtilsTest.testUnicodeSupport ____________________________ testFetchJobStoreFiles ____________________________ [gw2] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 340, in from_call result: Optional[TResult] = func() ^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 240, in lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 513, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 139, in _multicall raise exception.with_traceback(exception.__traceback__) File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 122, in _multicall teardown.throw(exception) # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/threadexception.py", line 87, in pytest_runtest_call yield from thread_exception_runtest_hook() File "/usr/lib/python3/dist-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook yield File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 122, in _multicall teardown.throw(exception) # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_call yield from unraisable_exception_runtest_hook() File "/usr/lib/python3/dist-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook yield File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 122, in _multicall teardown.throw(exception) # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/logging.py", line 849, in pytest_runtest_call yield from self._runtest_for(item, "call") File "/usr/lib/python3/dist-packages/_pytest/logging.py", line 832, in _runtest_for yield File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 122, in _multicall teardown.throw(exception) # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/capture.py", line 883, in pytest_runtest_call return (yield) ^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 122, in _multicall teardown.throw(exception) # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/skipping.py", line 256, in pytest_runtest_call return (yield) ^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 103, in _multicall res = hook_impl.function(*args) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 182, in pytest_runtest_call raise e File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 172, in pytest_runtest_call item.runtest() File "/usr/lib/python3/dist-packages/_pytest/python.py", line 1779, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 513, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 139, in _multicall raise exception.with_traceback(exception.__traceback__) File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 103, in _multicall res = hook_impl.function(*args) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/_pytest/python.py", line 195, in pytest_pyfunc_call result = testfunction(**testargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/toilDebugTest.py", line 118, in testFetchJobStoreFiles job_store_dir = workflow_debug_jobstore() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/toilDebugTest.py", line 32, in workflow_debug_jobstore subprocess.check_call( File "/usr/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['python3.11', '/build/reproducible-path/toil-6.1.0/src/toil/test/utils/ABCWorkflowDebug/debugWorkflow.py', '/tmp/tmpixco0diq/toilWorkflowRun']' returned non-zero exit status 1. ----------------------------- Captured stderr call ----------------------------- [2024-04-30T12:00:56-1200] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. Traceback (most recent call last): File "/build/reproducible-path/toil-6.1.0/src/toil/test/utils/ABCWorkflowDebug/debugWorkflow.py", line 181, in toil.start(job0) File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'initialize_jobs' kind-initialize_jobs/instance-j6tj_7hg v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. _________________________ PathIndexingPromiseTest.test _________________________ [gw1] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/promisesTest.py", line 69, in test self.assertEqual(Job.Runner.startToil(root, options), ('b', 43, 3)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'd' kind-d/instance-1e31aw_y v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.promisesTest.PathIndexingPromiseTest:test (Apr 30 2024 12:01:06:939505 PDT) ------------------------------ Captured log call ------------------------------- INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.promisesTest.PathIndexingPromiseTest.test ________________________ HelloWorldTest.testHelloWorld _________________________ [gw3] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/helloWorldTest.py", line 23, in testHelloWorld Job.Runner.startToil(HelloWorld(), options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 564, in _processReadyJob self._runJobSuccessors(job_id) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 452, in _runJobSuccessors self.issueJobs(successors) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 936, in issueJobs self.issueJob(job) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'FollowOn' kind-FollowOn/instance-capvs_sr v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.helloWorldTest.HelloWorldTest:testHelloWorld (Apr 30 2024 12:01:01:430013 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.helloWorldTest.HelloWorldTest.testHelloWorld ... DEBUG toil.common:common.py:665 Initialized default configuration: {"config_version": 1.0} DEBUG toil.batchSystems.options:options.py:188 Add options for single_machine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for grid_engine batch system DEBUG toil.batchSystems.options:options.py:188 Add options for lsf batch system DEBUG toil.batchSystems.options:options.py:188 Add options for slurm batch system DEBUG toil.batchSystems.options:options.py:188 Add options for torque batch system DEBUG toil.resource:resource.py:433 Module dir is /build/reproducible-path/toil-6.1.0/src, our prefix is /usr, virtualenv: False INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.leader:leader.py:923 Issued job 'HelloWorld' kind-HelloWorld/instance-gw4dtg82 v1 with job batch system ID: 1 and disk: 2.8 Gi, memory: 97.7 Ki, cores: 1, accelerators: [], preemptible: False INFO toil.leader:leader.py:1004 1 jobs are running, 0 jobs are issued and waiting to run INFO toil.leader:leader.py:923 Issued job 'childFn' kind-childFn/instance-b7iw2hix v1 with job batch system ID: 2 and disk: 2.8 Gi, memory: 976.6 Ki, cores: 1, accelerators: [], preemptible: False INFO toil.test:__init__.py:120 Tore down toil.test.src.helloWorldTest.HelloWorldTest.testHelloWorld ______________________ CachedUnpicklingJobStoreTest.test _______________________ [gw3] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/promisesTest.py", line 28, in test Job.Runner.startToil(root, options) File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'parent' kind-parent/instance-umho8hsx v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.promisesTest.CachedUnpicklingJobStoreTest:test (Apr 30 2024 12:01:17:938969 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.promisesTest.CachedUnpicklingJobStoreTest.test ... INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.promisesTest.CachedUnpicklingJobStoreTest.test _______________________ ChainedIndexedPromisesTest.test ________________________ [gw3] linux -- Python 3.11.9 /usr/bin/python3.11 Traceback (most recent call last): File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor yield File "/usr/lib/python3.11/unittest/case.py", line 623, in run self._callTestMethod(testMethod) File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod if method() is not None: ^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/test/src/promisesTest.py", line 45, in test self.assertEqual(Job.Runner.startToil(root, options), 42) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/job.py", line 2228, in startToil return toil.start(job) ^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 915, in start return self._runMainLoop(rootJobDescription) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/common.py", line 1396, in _runMainLoop jobCache=self._jobCache).run() ^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 257, in run self.innerLoop() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 755, in innerLoop self._processReadyJobs() File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 648, in _processReadyJobs self._processReadyJob(message.job_id, message.result_status) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 542, in _processReadyJob self.issueJob(readyJob) File "/build/reproducible-path/toil-6.1.0/src/toil/leader.py", line 913, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode, job_environment=job_environment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 758, in issueBatchJob self.check_resource_request(scaled_desc) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 509, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/singleMachine.py", line 505, in check_resource_request super().check_resource_request(requirer) File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 362, in check_resource_request raise e File "/build/reproducible-path/toil-6.1.0/src/toil/batchSystems/abstractBatchSystem.py", line 355, in check_resource_request raise InsufficientSystemResources(requirer, resource, available) toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job 'a' kind-a/instance-y33ff37a v1 is requesting 2147483648 bytes of memory, more than the maximum of 2109472768 bytes of memory that SingleMachineBatchSystem was configured with, or enforced by --maxMemory. Scale is set to 1. Scale is set to 1. Scale is set to 1. Scale is set to 1. ---------------------------- Captured stdout setup ----------------------------- [TEST] toil.test.src.promisesTest.ChainedIndexedPromisesTest:test (Apr 30 2024 12:01:20:915685 PDT) ------------------------------ Captured log call ------------------------------- INFO toil.test:__init__.py:115 Setting up toil.test.src.promisesTest.ChainedIndexedPromisesTest.test ... INFO toil:__init__.py:438 Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host wbq0. INFO toil.test:__init__.py:120 Tore down toil.test.src.promisesTest.ChainedIndexedPromisesTest.test =========================== short test summary info ============================ FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testDeleteLocalFile FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCachHitFileFromJobStore FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCacheMissFileFromJobStoreWithCachingReadFile FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadCacheMissFileFromJobStoreWithoutCachingReadFile FAILED src/toil/test/src/jobTest.py::JobTest::testDAGConsistency - toil.batch... FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testSimultaneousReadsUncachedStream FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testToilIsNotBroken FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility FAILED src/toil/test/src/jobTest.py::JobTest::testStatic2 - toil.batchSystems... FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteLocalFileToJobStore FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testToilIsNotBroken FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteNonLocalFileToJobStore FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility FAILED src/toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions FAILED src/toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions FAILED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testCachingFileStore FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedToilWF FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testPrintJobLog - Asserti... FAILED src/toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunction FAILED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStore FAILED src/toil/test/src/busTest.py::MessageBusTest::test_restart_without_bus_path FAILED src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testNonCachingFileStore FAILED src/toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointNotRetried FAILED src/toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testEncapsulation FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testRestartAttribute - As... FAILED src/toil/test/src/resumabilityTest.py::ResumabilityTest::test_chaining FAILED src/toil/test/utils/utilsTest.py::UtilsTest::testUnicodeSupport - toil... FAILED src/toil/test/utils/toilDebugTest.py::testFetchJobStoreFiles - subproc... FAILED src/toil/test/src/promisesTest.py::PathIndexingPromiseTest::test - toi... FAILED src/toil/test/src/helloWorldTest.py::HelloWorldTest::testHelloWorld - ... FAILED src/toil/test/src/promisesTest.py::CachedUnpicklingJobStoreTest::test FAILED src/toil/test/src/promisesTest.py::ChainedIndexedPromisesTest::test - ... =========== 34 failed, 132 passed, 349 skipped in 155.26s (0:02:35) ============ E: pybuild pybuild:389: test: plugin custom failed with: exit code=1: HOME=/build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil python3.11 setup.py develop --user && PYTHONPATH=/build/reproducible-path/toil-6.1.0/src:$PYTHONPATH PATH=/build/reproducible-path/toil-6.1.0/.pybuild/cpython3_3.11_toil/.local/bin/:$PATH TOIL_TEST_QUICK=True TOIL_SKIP_DOCKER=True python3.11 -m pytest -n auto --dist loadscope -vv -W ignore --ignore src/toil/test/lib/aws/test_s3.py --ignore src/toil/test/provisioners/aws/awsProvisionerTest.py --ignore src/toil/test/wdl/wdltoil_test.py --ignore src/toil/test/cwl/cwlTest.py --ignore src/toil/test/src/promisedRequirementTest.py --ignore src/toil/test/lib/test_ec2.py --ignore src/toil/test/batchSystems/batchSystemTest.py --ignore src/toil/test/lib/aws/test_iam.py -k "not (test_bioconda or test_run_conformance or testImportFtpFile or ToilWdlIntegrationTest or SortTest or testCwlexample or testVirtualEnv or ToilDocumentationTest or test_cwl_toil_kill or testImportReadFileCompatibility or CleanWorkDirTest or DeferredFunctionTest or WdlToilTest or test_cactus_integration)" /build/reproducible-path/toil-6.1.0/src/toil/test dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.12 3.11" returned exit code 13 make[1]: *** [debian/rules:25: override_dh_auto_test] Error 25 make[1]: Leaving directory '/build/reproducible-path/toil-6.1.0' make: *** [debian/rules:11: binary] Error 2 dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2 I: copying local configuration E: Failed autobuilding of package I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/21397 and its subdirectories Wed May 1 00:02:15 UTC 2024 W: No second build log, what happened?