there was already a system within gnome called
'bug buddy' to do something similar
. It would take the stack trace after
a crash and supposedly input this into the gnome
bug tracking system. Only 2 problems. 1. it required
you to install sendmail (or equiv) thus making
it completely useless to a large portion of the
people who would otherwise send in bugs, and
2. the people running gnome did not like bug-buddy.
i asked why, on irc, and they blew me off.
The bug buddy author came a little discouraged and went
on to work on other more "interesting" stuff
in gnome. He rejected my patch several times to make bug-buddy
not need sendmail anymore so i gave up on it.
Part of the problem is that most
programs depend on 15+ libraries, and
most of those libraries will not have debugging
symbols, so your stack trace isnt going
to be all that great.
But also i cannot
help but think there is a resistance among
open source project leaders to suddenly have
1500 crash bugs with stacktraces show up in their
In essence there are simply
not enough 'core developers' to wade through and fix
ten thousand obscure crashes that happen on suzie random
snowflakes system, it is simply easier to fix 'major' crashes
on the major distributions, and ignore the hordes of
people who are not linux-zealot enough to stick
with the program even though it bombs out. You can see
this time and time again in every open source
development conversation from irc to the
email lists. Developers will blame the crashes
on the supporting libraries, the underlying
hardware, etc, and so declare it "not their problem."
They dont want to program a workaround into
their program when 'the other guy should have done it
right in the first place'. Or they will declare the
crash a 'rare case' and just decide to ignore it.
The bulk of the conversations on open source
mailing lists are not about fixing crashes, they
are tangents about 'cool features' people want added
or pie-in-the-sky prognostications about what is
possible by using this or that library or this
or that structure or adopting so and so's work.
This brings me to the blatant and obvious
poverty of automated anti-bug tools on linux.
First of all, there is no decent easy to use
memory leak checker. Sure, memprof is not bad,
mpatrol is ok once you get past the mind
destorying weirdo setup process,
but mostly it will give you the results of libraries
that have no debugging symbols, so the bugs
are harder to track down. And trying to get
versions of shared libraries with debugging symbols
included, well, it usually means compiling these
libraries yourself. And with programs having
15+ libraries, all of whom might have memory leaks,
you can see that things get pretty hairy pretty quickly,
especially if you are trying to use the CVS updated
version of the first progarm, which depends on
CVS versions of 5 different libraries, which
might not even compile..... This is not helped
by the fact that the unix shared library linking
technology has more 'congealed' in an undocumented form
over the years than ever been designed (to plagiarize douglas
adams talking about vogon ships.)
Then there are the source level debuggers.
GDB is lovely, DDD is fine, kdb is jolly, when
they arent crashing and bringing your whole
And these tools are almost invariably way behind
the new technology of the languages and systems
they are trying to debug. For example,
gdb had no support for threads for a long time.
I doubt it has support for C++ STL templates,
instead spitting out 5 page long jibberish like
it did a year or so ago.
If you are working with some library like
QT or Gnome, similar problems occur, trying to
print out the values of variables such as
GtkWidgets or whatever.
Then there are the 'test suite frameworks' or
lack thereof. They often are complicated
and frustrating to use. Take DejaGNU for example.
Sure eventually you could
figure out how to use it properly, but the very nature of 'test suite' stuff
is that it needs to be as easy as hell,
just like internationalization of text strings
in GNU (the _() function around every "")
otherwise programmers are going to blow you off
add their feature and say 'screw you, automated testing
is a waste of time'.
Dont even get me started on how you are supposed
to automatically testsuite a GUI toolkit
program like gnome or KDE. windows programmers
can buy tools to do automated testing
of gui interfaces, but the average linux programmer
does not have this advantage.
help you if you have an 'average' machine,
these debugging tools typically take up
huge amounts of speed and memory. and since
you are running the 'debug version' of the
program it will take up at least twice
as much RAM and run much slower than usual.
Is this the fault of these tools' authors?
Partly, but partly it is that people who
bring out new technology for linux/OSS dont
give a rats ass about how anyone will debug it.
Debugging, for the author of the new gui
toolkit or the new framebuffer library or the
new threading system, is an afterthought,
"someone elses problem." But partly it
is because there is very little demand
for such tools by OSS project leaders.
Just run a memory leak checker on your favorite
open source program, especially gnome programs,
see how many improperly
destroyed memory allocations there are, and
realize that every one of those could potentially
lead to a crash, and then see the general attitude
of open source project leaders towards checking for memory leaks,
crashes, etc, on the mailing lists. It can take 50 hours to find
a single memory leak, and when it is fixed there
is absolutely 0 appreciable change in the way
the program works. The benefit will only be
felt indirectly 'down the road' when the thing doesnt crash,
but that is hard to quantify. Joe blow developer
is going to want to put that 50 hours into
something else with a more immediate reward,
like reworking the entire API,
moving to a new version, putting his favorite
pet feature in, or whatever, and naturally
this is what gets talked about most in
open source programming discussion groups.
The idea that you could analyze how you debug
things is a bit of a heresy, automating
such a process would be against the whole
philosophy of total benevolent dictatorship
of the project leader.
Supposedly in OSS 'all bugs are shallow' because
you have lots of people looking at them. But
the way it actually works out is that this
The extremely high 'barrier of entry' for someone
to fix a crash bug (download cvs trees, figure
out which cvs branch to use, find out which
dependencies there are, install this branch
alongside 'stable' versions so box is usable,
recompile bunches of things that may or may not
work, figure out who to talk to, try to figure
out how to communicate with them, work around
undocumented features that are necessary to even
get it to compile) virtually
guarantees that this 'labor pool' of people
who can eliminate crash bugs will remain small.
You dont theoretically
need a 'long term committed developer' nor to learn
every nook and cranny of a project to just
fix a crash bug, but it winds up happening
that due to the overhead and learning curve
to get in on the project, the resistance
to outsiders of the ledaers, the anti-democratic
philosophy inherent in most OSS projects,
etc, anyone trying to fix
a crash has to put as much time into overcoming
the red tape of the project as they do to
finding and fixing the crash. From a potential
crash fixers perspective, it just seems not worth it.
From the project leaders' perspective, they dont
want to focus all their time on a bunch of obscure
pathces from strangers, they want to deal
with their other project gurus and put interesting
things in that will directly benefit themselves
and their immediate peers.
In general, open source is very 'manual labor' intensive,
people like Linus continually decry trying
to take their job away with all this horrible
'automation' that 'cant do as good a job as a human being'.
Meanwhile commercial software companies are buying expensive
analysis tools like rational roses' memory
checkers and profilers... which people at
Redhat actually use according to what i heard
on IRC.... oh well.
I think you are fighting an uphill battle man,
one that I pretty much gave up on a long time ago.