Build failures with high parallel make(1) jobs with GCC

[Glen Barber]

I have been pounding my head against the desk for longer than I care to
admit with this failure.

I see this with powerpc, powerpc64, and now ia64. I initially thought
it was specific to powerpc{,64}, but now realize ia64 is also affected.

This build was running with -j48 on a 48-core machine, when the
following caused build failure:

=======================================================================

/usr/obj/ia64.ia64/usr/src/tmp/usr/bin/ld: cannot find -lm
--- libstdc++.so.6 ---
*** [libstdc++.so.6] Error code 1

make[4]: stopped in /usr/src/gnu/lib/libstdc++
A failure has been detected in another branch of the parallel make

=======================================================================

It is unclear to me when exactly this started happening, but it seems at
least two weeks is a reasonable estimate.

I realize this is not an entirely large chunk of useful information
regarding the build failure, but I have determined it is entirely
reproducible. I have determined that the failure case seems to
disappear with make(1) jobs <= 10, at least for powerpc.

The last successful build for powerpc on head/ was April 8. But I am
having trouble tracking down what commits may (or may not) have
contributed to recent high-parallel build failures.

Glen

[Ian Lepore]

Post by Glen Barber
I have been pounding my head against the desk for longer than I care to
admit with this failure.
I see this with powerpc, powerpc64, and now ia64. I initially thought
it was specific to powerpc{,64}, but now realize ia64 is also affected.
This build was running with -j48 on a 48-core machine, when the
=======================================================================
/usr/obj/ia64.ia64/usr/src/tmp/usr/bin/ld: cannot find -lm
--- libstdc++.so.6 ---
*** [libstdc++.so.6] Error code 1
make[4]: stopped in /usr/src/gnu/lib/libstdc++
A failure has been detected in another branch of the parallel make
=======================================================================
It is unclear to me when exactly this started happening, but it seems at
least two weeks is a reasonable estimate.
I realize this is not an entirely large chunk of useful information
regarding the build failure, but I have determined it is entirely
reproducible. I have determined that the failure case seems to
disappear with make(1) jobs <= 10, at least for powerpc.
The last successful build for powerpc on head/ was April 8. But I am
having trouble tracking down what commits may (or may not) have
contributed to recent high-parallel build failures.
Glen

A couple weeks corresponds somewhat with the parallel subdir build
changes (it's about 3 weeks now). Try this patch I cooked up today for
$work, and in src/lib/Makefile add .WAIT (as if it were a directory
name) between ${SUBDIR_ORDERED} and the rest of the directories.

-- Ian

[Glen Barber]

Post by Ian Lepore

Post by Glen Barber
The last successful build for powerpc on head/ was April 8. But I am
having trouble tracking down what commits may (or may not) have
contributed to recent high-parallel build failures.

A couple weeks corresponds somewhat with the parallel subdir build
changes (it's about 3 weeks now). Try this patch I cooked up today for
$work, and in src/lib/Makefile add .WAIT (as if it were a directory
name) between ${SUBDIR_ORDERED} and the rest of the directories.

The patch fails to apply cleanly, but as far as I can tell, it is due to
whitespace.

I'll hand-patch it, and report back.

Thanks.

Glen

[Glen Barber]

Post by Glen Barber

Post by Ian Lepore

Post by Glen Barber
The last successful build for powerpc on head/ was April 8. But I am
having trouble tracking down what commits may (or may not) have
contributed to recent high-parallel build failures.

A couple weeks corresponds somewhat with the parallel subdir build
changes (it's about 3 weeks now). Try this patch I cooked up today for
$work, and in src/lib/Makefile add .WAIT (as if it were a directory
name) between ${SUBDIR_ORDERED} and the rest of the directories.

The patch fails to apply cleanly, but as far as I can tell, it is due to
whitespace.
I'll hand-patch it, and report back.

Nope, I'm getting conflicts on revisions as far back as r251749.

Glen

[Ian Lepore]

Post by Glen Barber

Post by Glen Barber

Post by Ian Lepore

Post by Glen Barber
The last successful build for powerpc on head/ was April 8. But I am
having trouble tracking down what commits may (or may not) have
contributed to recent high-parallel build failures.

A couple weeks corresponds somewhat with the parallel subdir build
changes (it's about 3 weeks now). Try this patch I cooked up today for
$work, and in src/lib/Makefile add .WAIT (as if it were a directory
name) between ${SUBDIR_ORDERED} and the rest of the directories.

The patch fails to apply cleanly, but as far as I can tell, it is due to
whitespace.
I'll hand-patch it, and report back.

Nope, I'm getting conflicts on revisions as far back as r251749.
Glen

Doh! I completely forgot that's against 8.2 that we use at work. I'll
re-spin it for -current.

-- Ina

[Ian Lepore]

Post by Glen Barber

Post by Glen Barber

Post by Ian Lepore

Post by Glen Barber
The last successful build for powerpc on head/ was April 8. But I am
having trouble tracking down what commits may (or may not) have
contributed to recent high-parallel build failures.

A couple weeks corresponds somewhat with the parallel subdir build
changes (it's about 3 weeks now). Try this patch I cooked up today for
$work, and in src/lib/Makefile add .WAIT (as if it were a directory
name) between ${SUBDIR_ORDERED} and the rest of the directories.

The patch fails to apply cleanly, but as far as I can tell, it is due to
whitespace.
I'll hand-patch it, and report back.

Nope, I'm getting conflicts on revisions as far back as r251749.
Glen

This one should work better. The lib/Makefile is included this time.

-- Ian

[Glen Barber]

Post by Ian Lepore

Post by Glen Barber

Post by Glen Barber

Post by Ian Lepore

Post by Glen Barber
The last successful build for powerpc on head/ was April 8. But I am
having trouble tracking down what commits may (or may not) have
contributed to recent high-parallel build failures.

A couple weeks corresponds somewhat with the parallel subdir build
changes (it's about 3 weeks now). Try this patch I cooked up today for
$work, and in src/lib/Makefile add .WAIT (as if it were a directory
name) between ${SUBDIR_ORDERED} and the rest of the directories.

The patch fails to apply cleanly, but as far as I can tell, it is due to
whitespace.
I'll hand-patch it, and report back.

Nope, I'm getting conflicts on revisions as far back as r251749.
Glen

This one should work better. The lib/Makefile is included this time.

This patch applies fine.

It seems to already have an effect, but I won't go so far to say it
works until I see the 'World build completed: ' message.

I'll follow up tomorrow once build is done.

Thanks!

Glen

[Ian Lepore]

Post by Glen Barber

Post by Ian Lepore

Post by Glen Barber

Post by Glen Barber

Post by Ian Lepore

Post by Glen Barber
The last successful build for powerpc on head/ was April 8. But I am
having trouble tracking down what commits may (or may not) have
contributed to recent high-parallel build failures.

A couple weeks corresponds somewhat with the parallel subdir build
changes (it's about 3 weeks now). Try this patch I cooked up today for
$work, and in src/lib/Makefile add .WAIT (as if it were a directory
name) between ${SUBDIR_ORDERED} and the rest of the directories.

The patch fails to apply cleanly, but as far as I can tell, it is due to
whitespace.
I'll hand-patch it, and report back.

Nope, I'm getting conflicts on revisions as far back as r251749.
Glen

This one should work better. The lib/Makefile is included this time.

This patch applies fine.
It seems to already have an effect, but I won't go so far to say it
works until I see the 'World build completed: ' message.
I'll follow up tomorrow once build is done.
Thanks!
Glen

I think the change to lib/Makefile was not sufficient. After looking at
the notes on dependenices at the top of the makefile, I think the
attached has some chance of helping. The bsd.subdir.mk changes are now
commited as r264822.

-- Ian

[Glen Barber]

Post by Ian Lepore
I think the change to lib/Makefile was not sufficient. After looking at
the notes on dependenices at the top of the makefile, I think the
attached has some chance of helping. The bsd.subdir.mk changes are now
commited as r264822.

Thanks, I'll let you know in a few hours when the builds that are in
progress now are finished.

Glen