Tuesday, December 2, 2014

Building gdb 7.8.1 on Mint 17

; You can check the configuration of your current gdb with:
gdb --configuration
This GDB was configured as follows:
   configure --host=x86_64-linux-gnu --target=x86_64-linux-gnu
             --with-gdb-datadir=/usr/share/gdb (relocatable)
             --with-jit-reader-dir=/usr/lib/gdb (relocatable)
             --with-python=/usr (relocatable)
             --with-separate-debug-dir=/usr/lib/debug (relocatable)

; Snag the gdb 7.8.1 tarfile
wget http://ftp.gnu.org/gnu/gdb/gdb-7.8.1.tar.xz

; untar, cd into gdb-7.8.1

; If you want python 2.7, install libpython2.7-dev instead.
sudo apt-get install libpython3.4-dev liblzma-dev

; Run configure
./configure --host=x86_64-linux-gnu --with-lzma --target=x86_64-linux-gnu --with-expat --with-python=/usr/bin/python3.4 --with-separate-debug-dir=/usr/lib/debug --with-gdb-datadir=/usr/share/gdb --with-jit-reader-dir=/usr/lib/gdb --with-system-gdbinit=/etc/gdb/gdbinit --without-guile

; And build it.
make -j 18

Monday, September 29, 2014

glibtop: This machine has 56 CPUs, 32 are being monitored

Fully building TF2 dedicated server was taking around 13 minutes, 10 seconds on this:

Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz
mikesart@mikesart-petra:~$ grep -c model.name /proc/cpuinfo

I tend to muck with the lower level libraries which meant I was spending quite a bit of my day waiting for these builds to finish.

TF2 dedicated server has:
 - 1347 cpp files
 - 13 DSOs (with 13 .dbg files)
 - 12 static libraries

I looked at several things: ccache, precompiled headers, distcc, unity builds. We are using ccache fulltime, but when I switch between debug / release or add a new define here and there, it wasn't helping as much as I'd like. Distcc meant I'd have to manage several other machines, and the unity builds seemed like a whole bunch of work also. The precompiled headers gave a decent win (down to around 7 minutes), but (from what I could tell), the .gch file really wants to be where the main header file lives. We do debug/release/client/dedicated server builds all out of the same tree and this just won't work.

So I got one of these:

(2) Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
mikesart@mikesart-mint:~/valvesrc/ValveGames/staging/game$ grep -c model.name /proc/cpuinfo

The build time is now 3 minutes, 30 seconds.

I tried creating a ramdisk and putting the entire source tree and tmp directory there, and the build time was still 3m30s (compared to building from a Samsung SSD 850 PRO 512GB). I think a decent chunk of time right now is going towards linking - so I'm going to investigate the gold linker and see if I can get a bit more time back from that.

We are currently using gcc 4.6.1 crosstools w/ glibc-2.11.1. I'm going to investigate newer versions of gcc and clang when I get more time to play with this stuff. Have some bugs and features before I get to play more - hopefully turnaround will go a bit faster now though...

Sunday, June 8, 2014

Gdb script to spew malloc backtraces

We've got an app where "perf top" is showing malloc being called a lot. Now we need to figure out who is calling it.

Couple of ideas are obviously to use tcmalloc or something like this:


However these are live game instances running on a server somewhere in Germany (for example), and it's not easy to restart and preload DSOs. People playing on that server tend to frown upon this behavior also. :)

Came up with the idea to try using gdb to connect, set a breakpoint and log the next X hundred of the callstacks to a logfile. Script is down below. I've tried it on glxspheres and it seems like it'll work. I'm sure others have done this - if so and you have any comments / suggestions, please let me know.

# Use via something like this:
#    gdb -x ~/bin/spewmallocscript.gdb --pid=$(pidof glxspheres64)

# Make sure we never get asked y/n
set pagination off

# Overwrite logfile
set logging overwrite on
set logging file /tmp/gdb_malloc_calls.txt
set logging on

printf "\n\nSetting malloc breakpoint\n"
break malloc

set $i=0
set $count=100

printf "Logging %d callstacks...\n\n", $count

# add command for malloc breakpoint
bt 16

if $i < $count
  printf "\ncall #%d:\n", $i
  set $i=$i+1
  set logging redirect off


# send output to log file only
set logging redirect on

Tuesday, June 3, 2014

Man (and less) colors...

I had the following set in my environment.

export LESS='-QRS'
export LESS_TERMCAP_mb=$'\E[01;31m'
export LESS_TERMCAP_md=$'\E[01;38;5;74m'
export LESS_TERMCAP_me=$'\E[0m'
export LESS_TERMCAP_so="$(printf 'rev\nbold\nsetaf 3\n' | tput -S)"
export LESS_TERMCAP_se="$(tput sgr0)"
export LESS_TERMCAP_us=$'\E[04;38;5;146m'
export LESS_TERMCAP_ue=$'\E[0m'

Made for nice colored man pages. Unfortunately, when I used the "env" command, environment variables following the LESS_TERMCAP_* variables got mucked up: underlined, etc.

So I unset them and now use this alias:

alias man="LESS_TERMCAP_mb=$'\E[01;31m' LESS_TERMCAP_md=$'\E[01;38;5;74m' LESS_TERMCAP_me=$'\E[0m' LESS_TERMCAP_se=$'\E[0m' LESS_TERMCAP_so=$'\E[48;5;3
m\E[38;5;0m' LESS_TERMCAP_ue=$'\E[0m' LESS_TERMCAP_us=$'\E[04;38;5;146m' man"

alias less="LESS_TERMCAP_mb=$'\E[01;31m' LESS_TERMCAP_md=$'\E[01;38;5;74m' LESS_TERMCAP_me=$'\E[0m' LESS_TERMCAP_se=$'\E[0m' LESS_TERMCAP_so=$'\E[48;5;3
m\E[38;5;0m' LESS_TERMCAP_ue=$'\E[0m' LESS_TERMCAP_us=$'\E[04;38;5;146m' less"

Not sure it's the best solution, but it seemed better than the lesskey stuff. Also found the prompt option and added that:

export LESS='-QRS --prompt="%f (%pb\%, %lmth line, %L lines)$ '

The final fairly useful thing from poking in the code is the LESS_TERMCAP_DEBUG environment variable. It highlights the section names which is pretty darn useful for figuring the colors out. Example:
mikesart@mikesart-petra:~/dev$ LESS_TERMCAP_DEBUG=1 man strstr

<ti><ks><cr>STRSTR(3) Linux Programmer's Manual

       strstr, strcasestr - locate a substring


       <md>#include<me> <md><string.h><me>

... strstr(3) line 1/43 (END) (press h for help or q to quit)<se><ce>

Wednesday, April 30, 2014

32-bit Valgrind Notes

1. When running ./configure to build valgrind, look for this line:

    checking for 32 bit build support... yes

You can also use --enable-only64bit or --enable-only32bit if you only care about a specific platform.

2. Adding "-d" to valgrind will spew out a bunch of debug information, including which tool it's launching. Something like this:

--4603:1:launcher no client specified, defaulting platform to 'amd64-linux'
--4603:1:launcher launching /usr/local/lib/valgrind/memcheck-amd64-linux

"valgrind --verbose" can also be useful, of course.

3. Archive of valgrind users mailing list is here:


4. If you see this error when valgrind'ing a 32-bit application:

valgrind:  Fatal error at startup: a function redirection
valgrind:  which is mandatory for this platform-tool combination
valgrind:  cannot be set up.  Details of the redirection are:
valgrind:  A must-be-redirected function
valgrind:  whose name matches the pattern:      strlen
valgrind:  in an object with soname matching:   ld-linux.so.2
valgrind:  was not found whilst processing
valgrind:  symbols from the object with soname: ld-linux.so.2
valgrind:  Possible fixes: (1, short term): install glibc's debuginfo
valgrind:  package on this machine.  (2, longer term): ask the packagers
valgrind:  for your Linux distribution to please in future ship a non-
valgrind:  stripped ld.so (or whatever the dynamic linker .so is called)
valgrind:  that exports the above-named function using the standard
valgrind:  calling conventions for this platform.  The package you need
valgrind:  to install for fix (1) is called
valgrind:    On Debian, Ubuntu:                 libc6-dbg
valgrind:    On SuSE, openSuSE, Fedora, RHEL:   glibc-debuginfo
valgrind:  Cannot continue -- exiting now.  Sorry.

You most likely need the 32-bit libc-dbg package. On Debian based systems this should fix it:

sudo apt-get install libc6-dbg:i386

Tuesday, April 29, 2014

Valgrind Notes

Couple notes.

1. Command line for running valgrind with vogl capturing glxspheres64 from the vogl_build/bin directory:

valgrind --tool=memcheck --leak-check=full --error-limit=no --trace-children=yes --time-stamp=yes --log-file=/tmp/blah.log -- ../../bin/steamlauncher.sh --amd64 --gameid ./glxspheres64

2. Found some good stuff. Also a few things like this... :)

                // Get some entropy from the heap.
                p[i] = vogl_malloc(65536 * (i + 1));
                if (p[i])
                    for (uint j = 0; j < 16; j++)
                        gen.update_obj_bits(reinterpret_cast<const uint64_t *>(p)[j]);

2. Adding --track-origins=yes to the command line slows Valgrind down quite a bit but can really help. It added the line in bold for this stack trace (which wasn't making sense until we got this hint):

Uninitialised byte(s) found during client check request                                                                                                                             
   at 0x5422873: vogl_trace_stream_start_of_file_packet::compute_crc() const (vogl_trace_stream_types.h:185)
   by 0x54227B1: vogl_trace_stream_start_of_file_packet::check_crc(unsigned int) const (vogl_trace_stream_types.h:231)
   by 0x5421EE8: vogl_trace_stream_start_of_file_packet::full_validation(unsigned int) const (vogl_trace_stream_types.h:242)
   by 0x5420CB7: vogl_trace_file_writer::open(char const*, vogl_archive_blob_manager*, bool, bool, unsigned int) (vogl_trace_file_writer.cpp:82)
   by 0x517BAC0: vogl_global_init() (vogl_intercept.cpp:799) 
   by 0x92E236F: pthread_once (pthread_once.S:103)
   by 0x517A970: vogl_entrypoint_prolog(gl_entrypoint_id_t) (vogl_intercept.cpp:865) 
   by 0x50B3382: vogl_glXChooseVisual(_XDisplay const*, int, int const*) (gl_glx_func_defs.inc:91640)
   by 0x50B3302: glXChooseVisual (gl_glx_func_defs.inc:91635)
   by 0x403C84: main (glxspheres.c:716)
 Address 0x59632bd is 149 bytes inside data symbol "_ZZL21get_vogl_trace_writervE19s_vogl_trace_writer"
 Uninitialised value was created by a stack allocation
   at 0x5536214: vogl::init_uuid() (vogl_uuid.cpp:53)

3. And finally, if that doesn't do it, you can use code like this to help even more:

      #include "memcheck.h"
      uintptr_t addr = VALGRIND_CHECK_MEM_IS_DEFINED(ptr, len);
      if (addr)
          printf("VALGRIND_CHECK_MEM failed: %p %u\n", ptr, len);
          printf("  addr = %p\n", (void *)addr);

Documentation for these markups (and much, much more) here:


Just grab valgrind.h and memcheck.h. We've checked them into the extlib/valgrind directory in vogl.

Thursday, April 24, 2014

Bash Symbols

Debugging an issue where our preloaded vogl shared object is crashing bash. These are the steps I did to get the bash symbols on Linux Mint 16:

echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list

# NOTE: Since I'm on Linux Mint (Petra) I then had to edit ddebs.list and change petra to saucy.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01

wget -q http://ddebs.ubuntu.com/dbgsym-release-key.asc

sudo apt-key add dbgsym-release-key.asc

sudo apt-get update

apt-cache policy bash

# Returns something like this:

> bash:
>   Installed: 4.2-5ubuntu3
>   Candidate: 4.2-5ubuntu3
> ...

sudo apt-get install bash-dbgsym=4.2-5ubuntu3

# Grab the source:

apt-get source bash

cd bash-4.2

# Untar the source (I use atool, feel free to use tar xf or whatever):

atool -x bash-4.2.tar.xz

# Now in gdb (or your .gdbinit) you can point to the bash source. For me:

directory /home/mikesart/src/bash-4.2/bash-4.2

# Here are a couple of good of good links for all this.




Thursday, January 9, 2014

QtCreator and Ninja warning/error parsing

We have a custom build binary that launches ninja and we couldn't get QtCreator to parse the warnings from that output. Errors/warnings show up, they just weren't being parsed. I finally got QtCreator building with symbols and tracked it down to this in makestep.cpp:

 267     if (m_useNinja)
 268         AbstractProcessStep::stdError(line);
 269     else
 270         AbstractProcessStep::stdOutput(line);

Ninja is processing stderr and regular makefiles are processing stdout. So I added this to our custom shell script and it's working now:

Command: .../bin/mkvogl.sh
Arguments: --amd64 --debug 3>&1 1>&2 2>&3

I guess if you ever find QtCreator isn't parsing your output, try swapping stderr and stdout. :)

If anyone wants to get QtCreator building release with symbols on something vaguely resembling 64-bit Linux Mint 16, this is what I wound up doing:

apt-get install: libgl1-mesa-dev libxml2-dev libglib2.0-dev libxslt1-dev libglib2.0-dev libgstreamer-plugins-base0.10-dev libgstreamer0.10-dev
diff --git a/qtcreator.pri b/qtcreator.pri
index 1750705..9be9c03 100644
--- a/qtcreator.pri
+++ b/qtcreator.pri
@@ -180,6 +180,9 @@ unix {

     RCC_DIR = $${OUT_PWD}/.rcc
     UI_DIR = $${OUT_PWD}/.uic

 win32-msvc* {
cd ~/dev/qt-creator/src
qmake -r

If there is a better way of achieving this, please let me know - I don't know much about qmake and couldn't find anything...

Wednesday, January 8, 2014

QtCreator projects

I've switched to using QtCreator 3.0 as my main editor. I'm really liking it (and FakeVim!), but one big issue we've run into is projects. What files are loaded, what defines are set, files not listed in our makefiles (.sh), listing include files in our makefiles just so they're in the project, etc. It also likes to blast .user files where you open your makefile and now we're dealing with getting mercurial or git to ignore those, etc.

I wrote the below which just finds everything under our vogl project directory with specified extensions. It also takes some patterns to remove files after the fact (possibly someone can tell me a more optimal way of doing this?).

Next I'm going to figure out how to get QtCreator to parse ninja build output. (Grumble, grumble. :)

In any case, throwing it up here in case someone might find it useful...

# VoglProj QtCreator cmake file.
# Do the following in the directory above your vogl enlistment:
#  ln -s vogl/bin/qtcreator/CMakeLists.txt
# Then open it up with QtCreator and you should be off and running.
cmake_minimum_required(VERSION 2.8)

# List of file extensions that we search for.
set(EXTLIST *.i *.sh *.inl *.inc *.txt *.vs *.vp *.frag *.vert *.py *.m *.c* *.h* *.S)

# Vogl directory.
    message("\nERROR: ${VOGL_DIR} does not exist. Please put this script one level up from your vogl enlistement.\n")
    message(FATAL_ERROR "Exiting...")

# Create list of vogl directorie plus extensions.
foreach(ext ${EXTLIST})
    list(APPEND GLOBSPEC ${VOGL_DIR}/${ext})

# Search for all the files.
file(GLOB_RECURSE vogl_srcs

# Macro to remove files based on regex pattern.
macro(RemoveSrcFiles pat)
    foreach(file ${vogl_srcs})
        if(file MATCHES ${pat})
            list(APPEND result ${file})
    set(vogl_srcs ${result})

# Remove all files under .git and .hg directories.

# Spew out all files we've found.
set(count 0)
foreach(file ${vogl_srcs})
    math(EXPR count "${count} + 1")

message("${count} files added.\n")

add_executable(VoglProj ${vogl_srcs})
set_target_properties(VoglProj PROPERTIES LINKER_LANGUAGE C)