Using botch to generate transition build ordersThu, 11 Jul 2013 16:25 categories: debian
In October 2012, Joachim Breitner asked me whether botch (well, it
didnt have a name back then) can also be used to calculate a build order for
recompiling ~450 haskell-packages with a new ghc version (it was probably
the 7.6.1 release) to upload them to experimental. What is still blocking this
ability is the inability of botch to directly read
*.dsc files instead of
having to rely on
Sources files. On the other hand (in case
there exists a set of
Sources files) it is now much easier to
use botch for such a use case for which it was not originally designed.
To demonstrate how botch can be used to calculate the build order for library
transitions, I wrote the script
executes the individual botch tools in the correct order with the correct
arguments. To validate the correctness of botch, I compared its output to the
order which ben produces for the same transitions.
create-transition-order.sh script is called with a ben query string and
optionally a snapshot.debian.org timestamp as its arguments. The script
relies on ben being installed because ben query strings cannot be translated
grep-dctrl query strings as
ben query splits fields which contain
comma separated values (like
Build-Depends) at the comma before
it searches for matching strings. Unfortunately, the
create-transition-order.sh script also currently relies on a yet unreleased
ben feature which allows
ben download to create a
ben.cache file. You can
track the progress of this feature as bug#714703. Creating a
file is necessary because some queries rely on an association between binary
and source packages which is not present in all packages in a
For example the haskell transition query makes use of this feature by
.source field in its query.
The result of these trials was, that botch produced nearly the same build order for nearly all transitions. The only differences are due to shortcomings of ben and botch.
For example, ben is not able to create an order for transitions where involved
packages form one or more cycles. A prominent example is the haskell
ghc itself is only built during step 15 after many other
packages which would have needed ghc to be compiled first. Botch solves this
problem by reducing all strongly connected components in a cyclic graph to a
single vertex before creating the order. This operation makes the graph acyclic
and creating a build order trivially. The only remaining problems can then be
found within the strongly connected components (for Haskell they are of maximum
size two) but the overall build order is correct.
On the other hand botch has no notion of packages which are affected by a transition and thus creates a build order which in some cases is longer than the one created by ben. When generating the interdependencies between packages, ben only considers those which are part of the transition. Botch on the other hand considers all dependency relationships. It would be simple to solve this issue in botch by removing unaffected packages from the dependency graph through edge contraction (an operation already used by botch for other tasks).
This exercise also let me find another bug in
dose3 where libdose would
not associate a binary package with a binNMU version with its associated source
package but instead report a version mismatch error. This problem was also
reported in the dose bug tracker.