PPC64LE RHEL8 3.x buildbot is timing out on test_gdb

[JelleZijlstra]

Buildbot https://buildbot.python.org/all/#/builders/559 has been failing since May 12 with a failure like this:

test_pycfunction (test.test_gdb.PyBtTests.test_pycfunction)
Timeout (0:15:00)!
Thread 0x00007fff97a24d90 (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/selectors.py", line 415 in select
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 2075 in _communicate
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 1207 in communicate
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 112 in run_gdb
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 229 in get_stack_trace
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 936 in test_pycfunction

It last passed on commit 718b132 and succeeded on a052be4, suggesting that the culprit is one of these commits:

• a052be4 PEP 688: Crash if __release_buffer__ is called while an exception is active #104371: check return value of calling mv.release (GH-104371: check return value of calling mv.release #104417)
• 5b8cd5a test_typing fails refleak tests #104415: Fix refleak tests for typing.ByteString deprecation (gh-104415: Fix refleak tests for typing.ByteString deprecation #104416)
• d50c37d Use hypothesis for testing the standard library, falling back to stubs #86275: Implementation of hypothesis stubs for property-based tests, with zoneinfo tests (GH-86275: Implementation of hypothesis stubs for property-based tests, with zoneinfo tests #22863)
• 45f5aa8 Implement and document PEP 669. #103082: Filter LINE events in VM, to simplify tool implementation. (GH-103082: Filter LINE events in VM, to simplify tool implementation. #104387)
• 19ee53d Split up _testcapimodule.c #93649: Split gc- and allocation tests from _testcapimodule.c (gh-93649: Split gc- and allocation tests from _testcapimodule.c #104403)
• b2c1b4d Make it possible to mark Argument Clinic args as unused #104389: Add 'unused' keyword to Argument Clinic C converters (gh-104389: Add 'unused' keyword to Argument Clinic C converters #104390)
• 15795b5 Isolate the _io extension module #101819: Prepare _io._IOBase for module state (gh-101819: Prepare _io._IOBase for module state #104386)

[encukou]

@vstinner is the PEP-11 powerpc64le maintainer.
Victor, do you need any help with this issue? I'm afraid I can't help too much, though -- I don't have a PPC64 box available.

[encukou]

Oh, sorry! It turns out that powerpc64le is listed twice in PEP-11, and in the other entry, I am listed as the maintainer.
Unfortunately I will need to step down from this role as I no longer have easy access to the hardware.

[vstinner]

It turns out that powerpc64le is listed twice in PEP-11, and in the other entry, I am listed as the maintainer.

Yeah, ppc64le with GCC is supported at Tier-2 level: it's the config of RHEL operating system. ppc64le with clang is only supported at Tier-3 level: best effort.

Buildbot https://buildbot.python.org/all/#/builders/559 has been failing since May 12 with a failure like this (...)

I made multiple test_gdb changes last year. Is the ppc64le buildbot still failing with timeout time to time? If yes, does someone have links to such recent failures?

[vstinner]

I made multiple test_gdb changes last year.

test_gdb.test_pycfunction() is running tons of tests in a single method. I splitted test_gdb.py into a test_gdb test package (multiple .py files): commit 8f324b7 of issue gh-109972.

If it still fails time to time, maybe we should simply increase the timeout. I noticed that some PPC64LE buildbot workers are way slower than other buildbot workers. I don't recall which ones.

[vstinner]

Oh, sorry! It turns out that powerpc64le is listed twice in PEP-11, and in the other entry, I am listed as the maintainer. Unfortunately I will need to step down from this role as I no longer have easy access to the hardware.

If you still care, you can get access to ppc64le machines from Fedora: https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintainers I can give you a SSH access the ppc64 machines maintained by Red Hat, you should be able to easily reproduce the issue in a few minutes (need to clone the repo, build Python with similar build options than the worker, and run the test). Otherwise, we can just downgrade ppc64le+gcc to Tier-2, that's ok.

[encukou]

Let's do both: downgrade to tier 3, but I'll keep an eye on stable buildbots, and use Fedora infra when I around get to investigating this issue.

[vstinner]

PPC64LE RHEL8 3.x buildbot is timing out on test_gdb

I lost track of this issue. Does the issue still occur time to time?

[encukou]

No.
And the buildbot logs from May are gone, so it wouldn't be easy to investigate this. I don't think it's worth it to keep the issue open.

[vstinner]

Ok, thanks for keeping track of that issue.