path: root/doc/talks/debconf5/testing-security.paper

# Securing the Testing Distribution #

Joey Hess, DebConf5, Helsinki Finland

----------------------------------------------------------------------------

## Introduction ##

Debian's unique "testing" distribution is a potentially useful distribution
for both Debian end users and Custom Debian Distributions, who need a
reasonably up-to-date but not unstable version of Debian. One of the
distribution's main downfalls in attracting users has always been the lack
of security support. Over the past year, efforts have begun to change that.

## The "testing" Distribution ##

Debian's "testing" distribution is intended to be an automatically
stabilised version of Debian unstable, which is, in theory, kept free
of most of the release critical bugs that block a Debian release, while
still containing reasonably up-to-date software. Testing is intended to be
suitable as the basis for a stable Debian release at all times.

Testing grew out of the difficulty of managing a release based on the
unstable distribution. Before the advent of testing, Debian releases began
by taking a snapshot of unstable, fixing any show-stopper bugs, testing it,
and eventually releasing the snapshot as stable. As Debian grew larger and
supported more architectures, this process did not scale well, freeze
periods became ever longer, and Debian releases more out of date and less
internally consistent. These problems led to the invention of the testing
distribution in 2000 by Anthony Towns.

Updated packages reach testing under the control of a program (`britney`)
which ensures that several conditions are always met before a package in
testing can be updated to a newer version:

  * The new version cannot have more release critical bugs than the
    version already in testing.
  * The new package has to have aged and been tested for 10 days in
    Debian's "unstable" distribution, to help ensure it has no serious bugs.
    However, this delay can be reduced to 5 or even 2 days if packages have
    an urgent reason to be updated.
  * The new version of the package must have been built successfully on
    all the architectures Debian plans to release.
  * All the dependencies of the new version must be satisfied by packages
    in testing.

Handing most of the day-to-day management of a linux distribution over to a
program is a bold and unique idea among linux distributions, and making it
work has involved some unique challenges, many of them beyond the scope of
this discussion. On the whole, though, the resulting distribution has been
useful for its intended purpose of serving as a base for stable Debian
releases.

But ever since it was introduced, testing has seemed an attractive
distribution for other purposes as well. It's seemed useful for those who
need newer software than stable, such as updated desktop software, but
cannot deal with the problems inherent in using unstable. One example is
installing Debian on hundreds of desktops at a company or school, another
is installing it on a single computer which one does not have much time to
maintain. A third example is a Custom Debian Distribution, such as
[Debian-Edu](http://skolelinux.org).

While testing is used in all of these ways, a nasty little problem has kept
it from being widely used, limiting its users to a few with enough manpower
to do a lot of additional work, and a few who would rather not admit they're
using it. That problem is the difficulty in securing this distribution.

## Problems With Securing Testing ##

There are many problems that keep testing from being secure, and most have
their roots in the same conditions that are used to automatically control
the upgrading of packages in testing. Indeed, every one of those conditions
causes its own problems that can keep a security update in unstable from
reaching testing in an appropriately fast time span for testing to be safe
and secure.

  * Most security problems are release critical bugs, but if the package in
    unstable has additional release critical bugs besides the security fix,
    these can hold a crucial security fix out of testing indefinitely.

    The only way to avoid this is by back-porting the security fixes to the
    package version that is in testing, but besides often being difficult
    and introducing new bugs, back-ports need an upload queue like
    testing-proposed-updates or testing-security, and Debian has not kept
    these queues functional between stable releases.
  * The built-in delays in propagation of packages to testing keep any
    security fix out of testing for at least a couple of days, and possibly
    longer if it was not marked as being of high urgency.

    Release managers are able to reduce these delays, but some delay still
    exists, and reducing the delay increases the risk of buggy packages
    entering testing, which has to be weighed against the security problem
    if testing is to meet its original purpose.
  * If a package fails to auto-build on even one architecture, or if an
    architecture's auto-builder is slow or backlogged, this will keep it
    from reaching testing.

    Release managers are able to force in a package on the architectures
    for which it's already built, but breaking the consistency of testing
    like this can cause other problems and must be done with care.
  * A new version of a package often has dependencies on other new
    package versions, and anything that delays those packages from reaching
    testing can delay a security fix.

    Unlike all the other problems listed above, no solution is known for
    this problem. It can be ameliorated by avoiding overly tight library
    dependencies, but not avoided.
  * Testing is based on unstable which is always in a state of flux and for
    which security is by no means guaranteed.

All of these problems seem so insurmountable that for years no-one dared to
try to work on making testing more secure. Recently though a team has
formed that intended to do that, although the problems above remain as
constraints that must be constantly worked around.

## The Testing Security Team ##

The testing security team was formed in the fall of 2004. It is an open
team which currently has a combined membership of half a dozen Debian
developers and users. 

Team members spend between a few hours, up to a day or two of work time per
week on tracking and fixing security issues in testing. This is a fairly
large ongoing time commitment for a volunteer group, and in a survey of
some of the participants, they gave some reasons for working on the
security of testing that are not common in Debian, as well as some that
will be familiar to many Debian developers:

  * One member works for a organization which is deploying Debian sarge
    on a large scale and needs to keep track of security issues anyway.
    
  * One member works for the Debian-Edu distribution, which is based on
    Debian sarge, had to provide its own security support for its prior 
    release (Skolelinux) which was based on Debian woody, and needs to deal
    with security holes in testing.
  
  * One member works on another derived distribution (Univention Corporate
    Server), which is not based directly on sarge, and needs to keep track
    of security issues and cherry-pick fixes from Debian, and finds tracking
    and fixing security holes in testing a useful way to do this.
  
  * One member wants to work on Debian as a whole, not just on maintaining
    some packages, and finds working on security interesting.
    
  * Several members are interested in helping sarge release.
  
  * Several members want testing to have security support since they
    perceive it to be a useful alternative to stable for some Debian users.
    
  * Several members pointed out that the testing security team also makes
    Debian unstable more secure.
    
  * Several members appreciate the openness and transparency of the testing
    security team, in which everything is done in public mailing lists, bug
    tracking systems, and code repositories; compared to the stable
    security team which does many things in secret.
    
  * At least one member thinks that "it's fun".

Half of the members of the team have day jobs that involve large scale
Debian deployments, or distributions derived from Debian, all based on
the testing distribution. This could explain why it took four years from 
the introduction of testing to the formation of this team; it took nearly
that long for such uses of testing to become both necessary and
conceivable, and the pressure of trying to do these things with testing has
been a powerful motivator for many team members. If that is the case then
we can hope to see the team grow as others find themselves in similar
positions.

The testing security team is designed to scale up well and be easy to grow.
All of the work is innately divisible; our database of potential security
issues allows members to claim issues to check, and fixing the issues also
parallelises nicely. Little communication is needed between team members;
indeed days go by with the only communication being automatic updates on
who has checked which issues and what holes have been fixed. 

The barriers for entry to the team are as low as possible; team members do
not need to be Debian developers, the team operates on public information
about security issues, so there are no confidentiality requirements. The
only significant requirement is that team members have the time and skills
to do the work.

Indeed one of the goals of the talk which this paper accompanies is to find
someone new in the audience who can join the team. If you are interested or
wish to contact the team, our mailing list is
<secure-testing-team@lists.alioth.debian.org>.

## Tracking Vulnerabilities ##

A large part of the day to day job of the testing security team is simply
keeping track of security issues as they are discovered and announced,
checking whether they affect testing, and tracking fixes as they make their
way from unstable to testing. Some of this has been automated, but a lot of
the work must still be done by hand, and this work occupies a large part of
the total time spent by the team on security.

Different team members monitor different sources of data about new security
holes, including the [bugtraq](http://www.securityfocus.com/archive/1) and
[full-disclosure](http://lists.grok.org.uk/full-disclosure-charter.html)
mailing lists, the Debian bug tracking system, security advisories from
Debian stable, and security advisories from distributions related to
Debian (such as Ubuntu) and not related (such as Gentoo).

Unlike Debian's stable security team, we do not have access to the
vendor-sec mailing list where unreleased security holes are disclosed for
coordinated release by many linux distributions. From time to time we may
be forwarded such information but then we have to keep it private until it
is released, so only one team member will typically know about it. Since
this makes it hard to work as a team, we prefer to work with disclosed
security issues.

Keeping all of these sources of security holes straight and correlating
holes that are often mentioned in multiple places can be challenging.
Luckily this is not a problem that is unique to this security team, and
there is already a solution: The security community is standardising on the
[Common Vulnerabilities and Exposures](http://www.cve.mitre.org/)
dictionary. CVE is a global way to identify security issues that may affect
multiple distribution or operating systems. Each new security issue is
assigned a CVE ID, and these IDs are used to identify the issue in
advisories, bug reports, changelogs, and so on.

A sample CVE ID is "CAN-2005-1263", which refers to a linux kernel core
dump privilege escalation. Dozens to hundreds of new CVE IDs are issued
each week for security holes in linux and many other operating systems and
other software, and we try to have all the new IDs checked within a day of
each update.

This is one of the most parallelisable parts of the work of the testing
security team, and also often one of the most tedious, as the majority of
CVE entries are for software not in Debian, and some CVEs lack much useful
information or are unlikely to really be security holes, and we have to
check them manually, one at a time. Debian contains so many packages and
there are enough different ways to write the name of a piece of software
that it can involve quite a bit of checking just to see if a given piece
of software is in the distribution. Also, many CVE items affect multiple
packages in Debian due to code duplication and static linking, which can
make checking much harder.

Debian developers can do three things to make the testing security team's
CVE tracking easier.

1. If a security hole does not yet have a CVE ID assigned, contact the
   team or someone else and work to get one assigned.
2. Always list CVEs in the subjects of bug reports about security issues.
3. Always list CVEs in the changelog for security fixes.

As of June 2005, the testing security team had checked 8600 unique CVEs,
of which 1054 affected Debian. This included retroactively checking all
CVEs issued since the release of Debian 3.0.

## Fixing Vulnerabilities ##

The other half of the job of the testing security team is to work with
developers to get vulnerabilities fixed. Once we have identified a new
security hole, we will ensure that there is a bug in the Debian BTS for it
and if the maintainer can use our help, will work to find a patch or new
upstream release that fixes the hole, and if necessary, team members who
are Debian developers will make a Non-Maintainer Upload to fix the problem
in unstable.

Once a package is fixed in unstable, we can automatically track its
progress into testing, and will work with the release managers and others
to deal with any issues that might keep it out of testing. Some of the
techniques we use to deal with problems that might hold a package out of
testing include:

 * Uploading a back-ported security fix to the testing-proposed-updates
   queue, if that queue is functional.
   
 * Fixing unrelated release critical but not security-related bugs that are
   blocking the security fix from reaching testing.
   
 * Getting hints added to move a package into testing more quickly if it
   was uploaded with too low an urgency.
 
 * Dealing with problems that cause failure to build from source on the
   auto-builders, or bringing auto-builder problems to the attention of the
   package maintainer and/or auto-builder administrators.

 * Getting hopelessly insecure packages removed from testing.

 * Making it easy for the release team to follow what issues are keeping
   security fixes out of testing, so they can use the above and other
   techniques. They often do a better job than the testing security team
   at pushing the fixes into testing.

One further step is needed in the vulnerability fixing process before most
people will consider testing to really be security supported; that is
uploading fixes immediately to a temporary repository (such as
security.debian.org), and posting advisories so users can know a fix is
available for a security hole.

The testing security team hopes to do this, but is hampered by lack of
access to security.debian.org and lack of auto-builder support for testing.
We hope that these infrastuctural issues will be resolved after the release
of sarge. If not, we do plan to work around them and try to offer a
security repository and advisories anyway. We consider this very important
as without these last two pieces, most users will still consider testing to
be insecure and not security supported, and will not take into account all
the work we are already doing.

## Results ##

Anyone deciding whether to use testing and rely on the security support
provided by the testing security team, or to instead use stable, or even
unstable, would like to see a comparison of how long it takes to fix holes
in each of these versions of Debian. 

For example, we could examine all security holes for which Debian Security
Advisories (DSAs) have been issued for stable in the past year, and check
to see how much longer it took the holes to be fixed in testing. Or we
could examine all holes that have been known to affect testing and compare
how long it took to get the hole fixed in unstable with how long it took to
get the same fix into testing.

But this kind of comparison of how well the fixing of security holes is
managed is often flawed. Some of the pitfalls include:

  * Security holes differ widely in their severity, ease of exploitation,
    how many systems are vulnerable, etc. Simply counting security holes
    is not a good metric, and rating security holes or selecting only
    "important" holes to concentrate on can be subjective.
    
    This becomes very complex when security holes can be combined in
    different ways to produce a working exploit, as is often the case with
    remote code execution and local root exploit holes.

  * A single security hole can affect one package, or dozens of packages,
    and these can be fixed at different times and need to be tracked
    separately. However, a security hole that affects a dozen packages may
    still impact fewer machines than a hole that affects a single package.
    
  * Not all security holes will affect all distributions, but this doesn't
    necessarily mean that the distribution with less security holes is more
    secure. Instead it can mean that fewer people are reporting holes in
    its code-base, for whatever reason, or that the same holes exist in its
    code-base, but are not obvious due to code reorganizations. Or fewer
    holes might exist, but be more more serious. Or expectations of what
    constitutes a security hole can vary.

    These problems often show up in comparisons of the security of Windows
    and Linux, and of Linux and other Unix clones.

  * While the testing security team has comprehensive lists of all holes
    we've identified to affect testing, and have checked every CVE ID
    since 2002, no equivalent lists exist for stable, and our lists may be
    incomplete for unstable.
    
    The list of stable DSAs is known to be an incomplete list of security
    holes that affect stable, because the stable security team triages the
    more important and/or easy to fix hole first, and may never find the time
    or reason to fix all holes.

    So even getting a list of holes that affect the distributions to
    compare is problematic, without repeating all the work that has been
    done for testing, for the other distributions.

The comparisons that follow should be read with the above caveats in mind.
The only foolproof way to compare the security of two systems is probably
to put honeypots installed with each up on the internet and watch what
happens to them..

### Comparing stable and testing ###

A very rough comparison can be done with data the testing security team
gathers on a daily basis. Since the beginning of this year, we have tracked
whether security holes are already fixed in testing or not when a DSA is
issued for the holes.

However, no good data exists for the date that each of these holes was
fixed in testing, so we cannot compare how much more quickly or slowly the
fixes occurred, only which distribution fixed it first. And all of the
caveats above apply; security holes in DSAs vary in severity; DSAs can
cover multiple packages or holes; multiple DSAs may be released for one
hole; and a list of DSAs is not a complete list of holes that were
discovered in this time period.

Of 112 DSAs issued between January and May of 2005, 56 (exactly 50%) were
not fixed in testing before the DSA was released; 37 (33%) were fixed in
testing first, and 19 (17%) did not affect software in testing.

### Comparing testing and unstable ###

To track which holes are not fixed in unstable yet, and which holes have
been fixed in unstable but have not reached testing, we use an
[automatically updated list](http://spohr.debian.org/~joeyh/testing-security.html).

This list includes a count of the numbers of each type of unfixed hole, and
comparing these numbers is a useful (if flawed) metric to see if there is
typically more delay associated with fixing a hole in unstable in the first
place, or with getting the fix accepted into testing.

Unfortunately historical data is not available for these numbers, however
they have been roughly equal during most of this year, which suggests that
on average approximately half of the time to fix a hole is spent in getting
the hole fixed in unstable, and half in getting the fix into testing.

One interesting consequence of this is that it might be a better use of the
testing security teams's time to focus more effort on getting holes fixed
in unstable, since this is a more tractable problem than working on issues
that can block a fix from reaching testing, and since this would benefit
users of both distributions. However, it's probably too soon to tell, since
the release team was aggressively working on pushing fixes into testing
during this time period because of the impending release of sarge, and that
has likely skewed the numbers.

### Further Comparisons ###

Kernel security holes are a class of hole that is especially interesting,
because it can affect users of all versions of Debian, and because these
holes can be easily categorised into a few sets:

* remote root exploits
* local root exploits
* remote code execution
* remote denial of service
* local denial of service

Gathering the data and comparing the security history of testing with that
of stable during the first six months of 2005 would be a worthwhile
approach to better characterising the relative security of stable and
testing.

Other comparisons would be useful, especially if they were based on better
data and perhaps performed by a statistician. Real-world data about how
Debian machines are exploited would be even more useful in determining the
relative value of the work done by Debian's stable and testing security
teams. 

## Conclusions ##

The mantra that "testing is insecure" has become less true over the past
year thanks to the work of the testing security team, but much work still
remains before users can count on the security of testing. The team needs
to grow to the point that it can begin issuing formal advisories for
security holes in testing, so that users can begin to take these security
efforts seriously.

This effort began on the sidelines as a not-quite-official Debian project,
with many contributors who are not Debian developers; it is already
becoming a more recognised part of Debian as users, developers and even 
release managers realise its usefulness, and it can be hoped that this will
grow into a formal security support for testing on the same par as Debian's
security support for stable.

---------------------------------------------------------------------------

Copyright 2005 by Joey Hess <joeyh@debian.org>. 

Licensed under the GNU General Public License.