Svadev mailing list

[Santosh Nagarakatte <santoshn AT cis.upenn.edu>]

There are two issues with not optimizing the runtime and making it
dependent on the llvm build instead of program compilation options.

1) As we discussed, it is the slowdown caused by llvm debug build of the
runtime with a program which is optimized completely.

2) Lets say there is a memory safety violation  in a program. To debug
such a bug the end user recompiles the program with  symbols i.e "-g "
and no optimizations. If we use the release version of the runtime, it
is extremely difficult to get any meaningful information about whats
happening in the runtime.
To get any meaningful debugging  information about the runtime and hence
the program, the end user has to compile llvm and the safecode
infrastructure in debug mode and then debug the program.  I think its
the second use case that encourages us to move towards decoupling  LLVM
compilation mode and how the runtime is compiled.

What we essentially need  is  the following:
1) build both debug mode of the runtime and release mode of the runtime
2) depending on tthe optimization option, the compiler driver should
automatically choose either the debug runtime or the optimized runtime.

On the performance issues, I am seeing atleast 20X  or more overhead
consistently across all SPEC benchmarks~(SPEC 2006, SPEC 2000, SPEC 95) 
with SafeCode in Release mode on an x86_64 Core 2 machine. I am running
the performance experiments and will send you a detailed performance
result graph once my cluster jobs complete.

These are the other few observations that I made while doing some
performance evaluation with SafeCode.

I am seeing false violations in some benchmarks :eg: Compress from Spec 95.

Global external arrays trigger many warnings and are not checked as they
have [0 x  some type] in the bitcode.
These need to be fixed using LTO. This issue of external global arrays
is something even SoftBound needs to address. We can probably take an
unified approach to address this.

On the LTO front, in my experience, LTO provides 15-20% performance on
an average on a wide suite of benchmarks with native compilation.
Without LTO, SoftBound separate compilation with gcc incurred around
20-30% slowdown. However, we realized inlining the checks is super
important irrespective whether we perform LTO or not .

On 12/14/2011 01:07 PM, John Criswell wrote:
> On 12/14/11 2:05 AM, Santosh Nagarakatte wrote:
>> Hi all,
>>
>> I noticed that SafeCode compiles the runtime in Debug mode when the
>> SafeCode compiler is compiled in Debug+Asserts mode.
>> However as the runtime is injected into the binary, this translates to
>> execution time overheads  for any binary compiled in Debug mode.
>> This implies that to generate optimized code with SafeCode, we need to
>> compile the compiler in Release mode.
>
> Yes, that is correct.  However, the 3x figure I measured with SAFECode
> for LLVM 2.6/2.7 was done using a Debug version of the runtime and
> without many of the special optimizations that we did in our SAFECode
> research papers.  At this speed, SAFECode was still able to do more
> accurate checking than Valgrind's memcheck tool while still being
> much, much faster (3x is still too slow for production-use memory
> safety, though).
>
> In other words, using a debug runtime seems to have been fast enough
> in the past when using SAFECode as a debug tool.
>
>>
>> This can be very cumbersome especially when we want to test
>> performance of the application with optimizations when the compilation
>> of the application is done with optimizations and compiler is compiled
>> in debug+asserts mode.
>
> We actually used to do what you're suggesting, but I found it
> cumbersome.  The problem is that the LLVM build system really isn't
> designed to use mismatched Debug/Release files, so I would either
> forget to compile the run-time library in Release mode, or the runtime
> library would be hard-coded to Release mode, requiring me to hack the
> Makefiles to enable Debug mode for debugging, or some other such issue.
>
> IMHO, we should keep it simple: Release builds should be Release
> builds, and Debug builds should be Debug builds.  If you want the
> fastest runtime possible, do a Release build.  With the ability to
> have multiple object trees compiling from a single source tree, this
> will only cost compile time.
>
> That said, I would not oppose a configure option or a variable on the
> "make" command line that enables the behavior you want.  I just think
> that that behavior is confusing and should be enabled explicitly by
> someone who wants it.
>
>>
>> For example SafeCode has 50X overhead for a  simple test when the
>> program is compiled with O3  and the compiler is in compiled in debug
>> mode.
>
> Do you know what the overhead is when SAFECode is compiled in Release
> mode?
>
> As I've stated before, SAFECode suffered a serious performance
> regression when we moved it to LLVM 3.0.  There are several possible
> culprits (that are probably working together, no less):
>
> 1) The refactoring to merge it into Clang has changed the optimization
> order.  SAFECode is instrumenting code before all link-time
> optimizations have been applied.  Previously, we ran SAFECode on
> whole-program bitcode files in which all optimization (including LTO)
> was done first and then SAFECode instrumented the program.
>
> 2) There are several optimizations turned off because the code is not
> sufficiently robust or has not been updated to LLVM 3.0 (most notably
> the type-safety check elimination pass and the Monotonic loop
> optimization pass).  We have become far pickier about what gets
> enabled by default.
>
> 3) We are not using the pool allocator run-time (which may have sped
> up malloc() and friends).
>
> 4) We have new checks on the C standard library functions which were
> not enabled in LLVM 2.6 and 2.7.
>
> 5) Incomplete load/store checks actually do something now whereas they
> were no-ops before (and thus never added to the code).
>
> 6) The run-time is not linked in as an LLVM bitcode file.  This
> prevents the inlining of the run-time checks.  This could be
> especially bad for the fastlscheck/exactcheck checks (which, when
> inlined, can sometimes be eliminated entirely by standard compiler
> optimizations).
>
> I doubt the runtime library is the only issue.
>
> -- John T.
>