svn commit: r216977 - in head/libexec/rtld-elf: amd64 i386

[Dimitry Andric]

Author: dim
Date: Tue Jan 4 20:51:28 2011
New Revision: 216977
URL: http://svn.freebsd.org/changeset/base/216977

Log:
On amd64 and i386, tell the compiler to refrain from generating SSE,
3DNow, MMX and floating point instructions in rtld-elf.

Otherwise, _rtld_bind() (and whatever it calls) could possibly clobber
function arguments that are passed in SSE/3DNow/MMX/FP registers,
usually floating point values. This can happen, for example, when clang
generates SSE code for memset() or memcpy() calls.

One symptom of this is sshd dying early on amd64 with "PRNG not seeded",
which is ultimately caused by libcrypto.so.6 calling RAND_add() with a
double parameter. That parameter is passed via %xmm0, which gets wiped
out by an SSE memset() in _rtld_bind().

Reviewed by: kib, kan

Modified:
head/libexec/rtld-elf/amd64/Makefile.inc
head/libexec/rtld-elf/i386/Makefile.inc

Modified: head/libexec/rtld-elf/amd64/Makefile.inc
==============================================================================
--- head/libexec/rtld-elf/amd64/Makefile.inc Tue Jan 4 20:38:52 2011 (r216976)
+++ head/libexec/rtld-elf/amd64/Makefile.inc Tue Jan 4 20:51:28 2011 (r216977)
@@ -1,5 +1,6 @@
# $FreeBSD$

+CFLAGS+= -mno-sse -mno-sse2 -mno-sse3 -mno-mmx -mno-3dnow -msoft-float
# Uncomment this to build the dynamic linker as an executable instead
# of a shared library:
#LDSCRIPT= ${.CURDIR}/${MACHINE_CPUARCH}/elf_rtld.x

Modified: head/libexec/rtld-elf/i386/Makefile.inc
==============================================================================
--- head/libexec/rtld-elf/i386/Makefile.inc Tue Jan 4 20:38:52 2011 (r216976)
+++ head/libexec/rtld-elf/i386/Makefile.inc Tue Jan 4 20:51:28 2011 (r216977)
@@ -1,5 +1,6 @@
# $FreeBSD$

+CFLAGS+= -mno-sse -mno-sse2 -mno-sse3 -mno-mmx -mno-3dnow -msoft-float
# Uncomment this to build the dynamic linker as an executable instead
# of a shared library:
#LDSCRIPT= ${.CURDIR}/${MACHINE_CPUARCH}/elf_rtld.x

[Dimitry Andric]

-mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3
see 'grep -R "\-no-sse" /usr/src'. maybe the sorting order should stay
consistent?

I copied the order from sys/conf/kern.mk, lines 69 and 70:

-mfpmath=387 -mno-sse -mno-sse2 -mno-sse3 -mno-mmx -mno-3dnow \
-msoft-float -fno-asynchronous-unwind-tables

I don't particularly care about the order, though. The order you found
seems to be the historical order. If people prefer to have the same
order everywhere, I'll fix it.

also what's the status of clang? will these flags make sure that newer
cpu extension won't be activated? i checked
contrib/llvm/tools/clang/include/clang/Driver/Options.td
-m3dnowa
-mssse3
-msse4a
-msse4
-msse4_1
-msse4_2
-maes
-mavx

The in-tree clang has support for these "-mno-xxx" options:

-mno_3dnow
-mno_3dnowa
-mno_aes
-mno_avx
-mno_constant_cfstrings
-mno_mmx
-mno_omit_leaf_frame_pointer
-mno_pascal_strings
-mno_red_zone
-mno_relax_all
-mno_soft_float
-mno_sse
-mno_sse2
-mno_sse3
-mno_sse4
-mno_sse4_1
-mno_sse4_2
-mno_sse4a
-mno_ssse3
-mno_thumb
-mno_warn_nonportable_cfstrings

so I think you just looked incorrectly. :)

since these extensions only get set in a hand full of files maybe special
cases for CC == clang can be added.

I don't think it's necessary. Maybe there could be one central
definition (say, in share/sys.mk) for the flags that disable all
'special' features, e.g. for amd64 and i386:

CFLAGS_CONSERVATIVE=-mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -msoft-float

Similar flags could be defined for other arches too.

and nobody objected, but nobody wanted to commit the patch unfortunately.

@@ -26,7 +26,6 @@ CFLAGS= -DBOOTPROG=\"gptboot\" \
-fno-unit-at-a-time \
-mno-align-long-strings \
-mrtd \
- -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 \
-DGPT \
-D${GPTBOOT_UFS} \
-DSIOPRT=${BOOT_COMCONSOLE_PORT} \

Ehm, this looks wrong, those options should not be removed for anything
in sys/boot. Boot programs should normally refrain from using any
advanced CPU instructions. What did you want to achieve with this
patch?

[Gleb Kurtsou]

Post by Dimitry Andric
Author: dim
Date: Tue Jan 4 20:51:28 2011
New Revision: 216977
URL: http://svn.freebsd.org/changeset/base/216977
On amd64 and i386, tell the compiler to refrain from generating SSE,
3DNow, MMX and floating point instructions in rtld-elf.
Otherwise, _rtld_bind() (and whatever it calls) could possibly clobber
function arguments that are passed in SSE/3DNow/MMX/FP registers,
usually floating point values. This can happen, for example, when clang
generates SSE code for memset() or memcpy() calls.

-mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3
see 'grep -R "\-no-sse" /usr/src'. maybe the sorting order should stay
consistent?
also what's the status of clang? will these flags make sure that newer
cpu extension won't be activated? i checked
contrib/llvm/tools/clang/include/clang/Driver/Options.td
-m3dnowa
-mssse3
-msse4a
-msse4
-msse4_1
-msse4_2
-maes
-mavx
since these extensions only get set in a hand full of files maybe special
cases for CC == clang can be added.

Why not to add NO_HWFLOAT knob (or similar) into makefile
infrastructure. And set CFLAGS accordingly, depending on CC, arch, etc.
These flags are getting rather common in tree.

and nobody objected, but nobody wanted to commit the patch unfortunately.
cheers.
alex

[John Baldwin]

Post by Gleb Kurtsou

Post by Dimitry Andric
Author: dim
Date: Tue Jan 4 20:51:28 2011
New Revision: 216977
URL: http://svn.freebsd.org/changeset/base/216977
On amd64 and i386, tell the compiler to refrain from generating SSE,
3DNow, MMX and floating point instructions in rtld-elf.
Otherwise, _rtld_bind() (and whatever it calls) could possibly clobber
function arguments that are passed in SSE/3DNow/MMX/FP registers,
usually floating point values. This can happen, for example, when clang
generates SSE code for memset() or memcpy() calls.

-mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3
see 'grep -R "\-no-sse" /usr/src'. maybe the sorting order should stay
consistent?
also what's the status of clang? will these flags make sure that newer
cpu extension won't be activated? i checked
contrib/llvm/tools/clang/include/clang/Driver/Options.td
-m3dnowa
-mssse3
-msse4a
-msse4
-msse4_1
-msse4_2
-maes
-mavx
since these extensions only get set in a hand full of files maybe special
cases for CC == clang can be added.

Why not to add NO_HWFLOAT knob (or similar) into makefile
infrastructure. And set CFLAGS accordingly, depending on CC, arch, etc.
These flags are getting rather common in tree.

It strikes me that we really want clang/gcc to have some sort of
'-mno-hwfloat' so we don't keep having to add new flags in the future.

[Dimitry Andric]

Post by John Baldwin

Post by Gleb Kurtsou
Why not to add NO_HWFLOAT knob (or similar) into makefile
infrastructure. And set CFLAGS accordingly, depending on CC, arch, etc.
These flags are getting rather common in tree.

It strikes me that we really want clang/gcc to have some sort of
'-mno-hwfloat' so we don't keep having to add new flags in the future.

This is not just about floats, clang can also use SSE/AVX instructions
for e.g. memset(), memcpy() and the like, or even for structure
assignments.

As I replied to Alexander, it's probably better to just put such "lowest
common denominator for an arch" flags into sys.mk or bsd.cpu.mk.

[David Schultz]

Post by John Baldwin

Post by Gleb Kurtsou

Post by Dimitry Andric
Author: dim
Date: Tue Jan 4 20:51:28 2011
New Revision: 216977
URL: http://svn.freebsd.org/changeset/base/216977
On amd64 and i386, tell the compiler to refrain from generating SSE,
3DNow, MMX and floating point instructions in rtld-elf.
Otherwise, _rtld_bind() (and whatever it calls) could possibly clobber
function arguments that are passed in SSE/3DNow/MMX/FP registers,
usually floating point values. This can happen, for example, when clang
generates SSE code for memset() or memcpy() calls.

-mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3
see 'grep -R "\-no-sse" /usr/src'. maybe the sorting order should stay
consistent?
also what's the status of clang? will these flags make sure that newer
cpu extension won't be activated? i checked
contrib/llvm/tools/clang/include/clang/Driver/Options.td
-m3dnowa
-mssse3
-msse4a
-msse4
-msse4_1
-msse4_2
-maes
-mavx
since these extensions only get set in a hand full of files maybe special
cases for CC == clang can be added.

Why not to add NO_HWFLOAT knob (or similar) into makefile
infrastructure. And set CFLAGS accordingly, depending on CC, arch, etc.
These flags are getting rather common in tree.

It strikes me that we really want clang/gcc to have some sort of
'-mno-hwfloat' so we don't keep having to add new flags in the future.

It would have to be called something like -mno-xmmregs. Many SSE
instructions are for vector operations on integers, not floating
point. Furthermore, I don't think the i387 floating point
instructions would cause the problem being addressed here
(although they're a bad idea for _rtld_bind() for other reasons).

[Warner Losh]

Post by John Baldwin

Post by Gleb Kurtsou

Post by Dimitry Andric
Author: dim
Date: Tue Jan 4 20:51:28 2011
New Revision: 216977
URL: http://svn.freebsd.org/changeset/base/216977
On amd64 and i386, tell the compiler to refrain from generating SSE,
3DNow, MMX and floating point instructions in rtld-elf.
Otherwise, _rtld_bind() (and whatever it calls) could possibly clobber
function arguments that are passed in SSE/3DNow/MMX/FP registers,
usually floating point values. This can happen, for example, when clang
generates SSE code for memset() or memcpy() calls.

-mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3
see 'grep -R "\-no-sse" /usr/src'. maybe the sorting order should stay
consistent?
also what's the status of clang? will these flags make sure that newer
cpu extension won't be activated? i checked
contrib/llvm/tools/clang/include/clang/Driver/Options.td
-m3dnowa
-mssse3
-msse4a
-msse4
-msse4_1
-msse4_2
-maes
-mavx
since these extensions only get set in a hand full of files maybe special
cases for CC == clang can be added.

Why not to add NO_HWFLOAT knob (or similar) into makefile
infrastructure. And set CFLAGS accordingly, depending on CC, arch, etc.
These flags are getting rather common in tree.

It strikes me that we really want clang/gcc to have some sort of
'-mno-hwfloat' so we don't keep having to add new flags in the future.

gcc already has -msoftfloat, but I guess that's a little different than
what you are proposing...

Warner

[John Baldwin]

Post by Dimitry Andric

Post by John Baldwin

Post by Gleb Kurtsou
Why not to add NO_HWFLOAT knob (or similar) into makefile
infrastructure. And set CFLAGS accordingly, depending on CC, arch, etc.
These flags are getting rather common in tree.

It strikes me that we really want clang/gcc to have some sort of
'-mno-hwfloat' so we don't keep having to add new flags in the future.

This is not just about floats, clang can also use SSE/AVX instructions
for e.g. memset(), memcpy() and the like, or even for structure
assignments.

Yes, but the thing that all these extensions have in common is that they use
FPU state (i.e. subject to DNA traps, managed via *SAVE and *RSTOR, etc.)
and that is the problem with using them in boot code or rtld. What I would
want a -mno-hwfloat flag to do is to disable use of anything that would
require working FPU state handling.

[Nathan Whitehorn]

Post by John Baldwin

Post by Dimitry Andric

Post by John Baldwin

Post by Gleb Kurtsou
Why not to add NO_HWFLOAT knob (or similar) into makefile
infrastructure. And set CFLAGS accordingly, depending on CC, arch, etc.
These flags are getting rather common in tree.

It strikes me that we really want clang/gcc to have some sort of
'-mno-hwfloat' so we don't keep having to add new flags in the future.

This is not just about floats, clang can also use SSE/AVX instructions
for e.g. memset(), memcpy() and the like, or even for structure
assignments.

Yes, but the thing that all these extensions have in common is that they use
FPU state (i.e. subject to DNA traps, managed via *SAVE and *RSTOR, etc.)
and that is the problem with using them in boot code or rtld. What I would
want a -mno-hwfloat flag to do is to disable use of anything that would
require working FPU state handling.

You would also want this to be cross-platform, in which case it's more
than floating point. E.g. on powerpc, you also want to disable both FP
and vector extensions, which use separate sets of instructions and
registers. I guess overriding CPU type to be something very old (386?)
potentially deoptimizes the code?
-Nathan

[John Baldwin]

Post by Nathan Whitehorn

Post by John Baldwin

Post by Dimitry Andric

Post by John Baldwin

Post by Gleb Kurtsou
Why not to add NO_HWFLOAT knob (or similar) into makefile
infrastructure. And set CFLAGS accordingly, depending on CC, arch, etc.
These flags are getting rather common in tree.

It strikes me that we really want clang/gcc to have some sort of
'-mno-hwfloat' so we don't keep having to add new flags in the future.

This is not just about floats, clang can also use SSE/AVX instructions
for e.g. memset(), memcpy() and the like, or even for structure
assignments.

Yes, but the thing that all these extensions have in common is that they use
FPU state (i.e. subject to DNA traps, managed via *SAVE and *RSTOR, etc.)
and that is the problem with using them in boot code or rtld. What I would
want a -mno-hwfloat flag to do is to disable use of anything that would
require working FPU state handling.

You would also want this to be cross-platform, in which case it's more
than floating point. E.g. on powerpc, you also want to disable both FP
and vector extensions, which use separate sets of instructions and
registers. I guess overriding CPU type to be something very old (386?)
potentially deoptimizes the code?

Hmm, I don't know if this is generically MI, but at least for x86 having a
single -mno-sse-xmm-mmx-anything-registers-please would be useful I think.

[Dimitry Andric]

Post by Nathan Whitehorn
You would also want this to be cross-platform, in which case it's more
than floating point. E.g. on powerpc, you also want to disable both FP
and vector extensions, which use separate sets of instructions and
registers.

Yeah, but in any case, hacking the compiler itself is not the proper way
to achieve this goal, IMHO. It should go into sys.mk, or more
appropriately bsd.cpu.mk. I'll make a crude diff for this tonight.

Post by Nathan Whitehorn
I guess overriding CPU type to be something very old (386?)
potentially deoptimizes the code?

That won't work for amd64. :)

[John Baldwin]

Post by Dimitry Andric

Post by Nathan Whitehorn
You would also want this to be cross-platform, in which case it's more
than floating point. E.g. on powerpc, you also want to disable both FP
and vector extensions, which use separate sets of instructions and
registers.

Yeah, but in any case, hacking the compiler itself is not the proper way
to achieve this goal, IMHO. It should go into sys.mk, or more
appropriately bsd.cpu.mk. I'll make a crude diff for this tonight.

I think this is a workaround for a deficiency in the compiler, and one we are
probably stuck with. But it sure would be nice to have in the compiler.

[Alexander Best]

Post by John Baldwin

Post by Dimitry Andric

Post by Nathan Whitehorn
You would also want this to be cross-platform, in which case it's more
than floating point. E.g. on powerpc, you also want to disable both FP
and vector extensions, which use separate sets of instructions and
registers.

Yeah, but in any case, hacking the compiler itself is not the proper way
to achieve this goal, IMHO. It should go into sys.mk, or more
appropriately bsd.cpu.mk. I'll make a crude diff for this tonight.

I think this is a workaround for a deficiency in the compiler, and one we are
probably stuck with. But it sure would be nice to have in the compiler.

ehm...i'm not an expert on clang, but why not ask the clang developers to add
a switch to disable all floating point extensions and for archs that use it
another switch to disable vector extensions? it seems the clang developers are
very open minded and open to new ideas.

cheers.
alex

Post by John Baldwin
--
John Baldwin

[Doug Barton]

judging from the discussion going on right now it seems those flags will be
grouped together to form a new variable. so things will probably change shortly
and fixing the order is probably not necessary.

Much better to fix the problem properly now than to rely on future work
that may or may not happen. I realize that you alluded to this later in
your message, but I think as a general principle this is worth reinforcing.

some people have proposed hacking into clang which i personally think is a very
bad idea. why not contact the clang developers? they might like the idea of a
switch disabling all advanced extensions for every architecture?

I agree with this. We have a very awkward situation right now with lots
of local hacks in our version of gcc that in an ideal world we would not
replicate with clang; particularly considering the much lower barrier to
entry when it comes to contributing things back.


Doug

[John Baldwin]

Post by Doug Barton

judging from the discussion going on right now it seems those flags will be
grouped together to form a new variable. so things will probably change shortly
and fixing the order is probably not necessary.

Much better to fix the problem properly now than to rely on future work
that may or may not happen. I realize that you alluded to this later in
your message, but I think as a general principle this is worth reinforcing.

some people have proposed hacking into clang which i personally think is a very
bad idea. why not contact the clang developers? they might like the idea of a
switch disabling all advanced extensions for every architecture?

I agree with this. We have a very awkward situation right now with lots
of local hacks in our version of gcc that in an ideal world we would not
replicate with clang; particularly considering the much lower barrier to
entry when it comes to contributing things back.

My suggestion was that we ask clang to add a '-mno-whatever' and hopefully we
could convince gcc to follow suit. clang developers seem to be fairly
receptive, so I was hoping one of our clang liaisons could suggest it. :)

[Dimitry Andric]

Post by John Baldwin
My suggestion was that we ask clang to add a '-mno-whatever' and hopefully we
could convince gcc to follow suit. clang developers seem to be fairly
receptive, so I was hoping one of our clang liaisons could suggest it. :)

These options already exist, e.g -mno-sse -mno-sse2 and so on. The
semantics of -mno-everything would be a bit hard to predict, especially
with new instruction sets coming out all the time. :)

[John Baldwin]

Post by Dimitry Andric

Post by John Baldwin
My suggestion was that we ask clang to add a '-mno-whatever' and hopefully we
could convince gcc to follow suit. clang developers seem to be fairly
receptive, so I was hoping one of our clang liaisons could suggest it. :)

These options already exist, e.g -mno-sse -mno-sse2 and so on. The
semantics of -mno-everything would be a bit hard to predict, especially
with new instruction sets coming out all the time. :)

That's actually the point. I would want the compiler to automatically add new
register set options to -mno-everything when it grew support for them so we
don't have to frob the Makefiles every time.

[Alexander Best]

Post by John Baldwin

Post by Doug Barton

judging from the discussion going on right now it seems those flags will be
grouped together to form a new variable. so things will probably change shortly
and fixing the order is probably not necessary.

Much better to fix the problem properly now than to rely on future work
that may or may not happen. I realize that you alluded to this later in
your message, but I think as a general principle this is worth reinforcing.

some people have proposed hacking into clang which i personally think is a very
bad idea. why not contact the clang developers? they might like the idea of a
switch disabling all advanced extensions for every architecture?

I agree with this. We have a very awkward situation right now with lots
of local hacks in our version of gcc that in an ideal world we would not
replicate with clang; particularly considering the much lower barrier to
entry when it comes to contributing things back.

My suggestion was that we ask clang to add a '-mno-whatever' and hopefully we
could convince gcc to follow suit. clang developers seem to be fairly
receptive, so I was hoping one of our clang liaisons could suggest it. :)

why gcc? even if they decide to add such a switch it will be gpl3'ed. shouldn't
gcc with clang at hand be considered legacy software?

cheers.
alex

ps: btw there is a patch in GNATS which bumps base gcc to the very last
revision which didn't include gplv3. i think the patch fixes quite a few
issues: 153298.

also it seems apple is maintaining a gcc branch which has a lot of
improvements, yet it is based on gcc 4.2.1 and thus licensed under gplv2.
i'm not sure if it is avalable somewhere, but having a peek at the changes they
made would in fact be nice. still i don't think tackling base gcc is worth the
hassle, since that time could be spent much better improving clang.

Post by John Baldwin
--
John Baldwin

[John Baldwin]

Post by Alexander Best

Post by John Baldwin

Post by Doug Barton

judging from the discussion going on right now it seems those flags will be
grouped together to form a new variable. so things will probably change shortly
and fixing the order is probably not necessary.

Much better to fix the problem properly now than to rely on future work
that may or may not happen. I realize that you alluded to this later in
your message, but I think as a general principle this is worth reinforcing.

some people have proposed hacking into clang which i personally think is a very
bad idea. why not contact the clang developers? they might like the idea of a
switch disabling all advanced extensions for every architecture?

I agree with this. We have a very awkward situation right now with lots
of local hacks in our version of gcc that in an ideal world we would not
replicate with clang; particularly considering the much lower barrier to
entry when it comes to contributing things back.

My suggestion was that we ask clang to add a '-mno-whatever' and hopefully we
could convince gcc to follow suit. clang developers seem to be fairly
receptive, so I was hoping one of our clang liaisons could suggest it. :)

why gcc? even if they decide to add such a switch it will be gpl3'ed. shouldn't
gcc with clang at hand be considered legacy software?

I think a more realistic view is that some folks will want to use a gpl3
toolchain (e.g. platforms clang doesn't support (or support well) which
need newer binutils, etc.). The gpl3 bits don't have to live in the tree,
but I think our build infrastructure should support using them as an
external toolchain.

[Alexander Best]

Post by John Baldwin

Post by Alexander Best

Post by John Baldwin

Post by Doug Barton

judging from the discussion going on right now it seems those flags will be
grouped together to form a new variable. so things will probably change shortly
and fixing the order is probably not necessary.

Much better to fix the problem properly now than to rely on future work
that may or may not happen. I realize that you alluded to this later in
your message, but I think as a general principle this is worth reinforcing.

some people have proposed hacking into clang which i personally think is a very
bad idea. why not contact the clang developers? they might like the idea of a
switch disabling all advanced extensions for every architecture?

I agree with this. We have a very awkward situation right now with lots
of local hacks in our version of gcc that in an ideal world we would not
replicate with clang; particularly considering the much lower barrier to
entry when it comes to contributing things back.

My suggestion was that we ask clang to add a '-mno-whatever' and hopefully we
could convince gcc to follow suit. clang developers seem to be fairly
receptive, so I was hoping one of our clang liaisons could suggest it. :)

why gcc? even if they decide to add such a switch it will be gpl3'ed. shouldn't
gcc with clang at hand be considered legacy software?

I think a more realistic view is that some folks will want to use a gpl3
toolchain (e.g. platforms clang doesn't support (or support well) which
need newer binutils, etc.). The gpl3 bits don't have to live in the tree,
but I think our build infrastructure should support using them as an
external toolchain.

good point. would be nice to be able to do
echo "CC = gcc46" >> /etc/src.conf

... expecially for benchmarking clang vs. a recent gcc for world/kernel.

cheers.
alex

Post by John Baldwin
--
John Baldwin

[Alexander Best]

Post by Doug Barton

judging from the discussion going on right now it seems those flags will be
grouped together to form a new variable. so things will probably change
shortly
and fixing the order is probably not necessary.

Much better to fix the problem properly now than to rely on future work
that may or may not happen. I realize that you alluded to this later in
your message, but I think as a general principle this is worth reinforcing.

this patch should make all the -mno-* flags be sorted conistently.

cheers.
alex

Post by Doug Barton

some people have proposed hacking into clang which i personally think is a
very
bad idea. why not contact the clang developers? they might like the idea
of a
switch disabling all advanced extensions for every architecture?

I agree with this. We have a very awkward situation right now with lots
of local hacks in our version of gcc that in an ideal world we would not
replicate with clang; particularly considering the much lower barrier to
entry when it comes to contributing things back.
Doug
--
Nothin' ever doesn't change, but nothin' changes much.
-- OK Go
Breadth of IT experience, and depth of knowledge in the DNS.
Yours for the right price. :) http://SupersetSolutions.com/