md5sums files

Submitted by Wouter Verhelst on Tue, 03/02/2010 - 19:07

Debian Development

Hello world,

wouter@celtic:/var/lib/dpkg/info$ ls *md5sums|wc -l
2340
wouter@celtic:/var/lib/dpkg/info$ ls *sums|wc -l
2340
wouter@celtic:/var/lib/dpkg/info$ dpkg -l|sed -e'1,/=====/d'|wc -l
2483

I must say I was somewhat surprised by these numbers. Out of 2483
packages installed on my laptop, 2340 install md5sums. While that
might've been useful at some point, I don't think it still is.

In this day and age of completely and utterly broken MD5[0], I think we
should stop providing these files, and maybe provide something else
instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing
md5sums.

Or is it useful to be able to say "if it doesn't check out, it's
certainly corrupt, and if it does check out, it may be corrupt"? Didn't
think so.

Thoughts?

[0] No reference. It's all over the internet. If you didn't know about
MD5 being broken yet, where have you been sleeping these past few years?

--
The biometric identification system at the gates of the CIA headquarters
works because there's a guard with a large gun making sure no one is
trying to fool the system.
http://www.schneier.com/blog/archives/2009/01/biometrics.html

md5sums files

Submitted by Loïc Minier on Wed, 03/03/2010 - 08:14.

On Wed, Mar 03, 2010, Wouter Verhelst wrote:
> In this day and age of completely and utterly broken MD5[0], I think we
> should stop providing these files, and maybe provide something else
> instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing
> md5sums.
>
> Or is it useful to be able to say "if it doesn't check out, it's
> certainly corrupt, and if it does check out, it may be corrupt"? Didn't
> think so.

I use them regularly after laptop crashes; I found some misinstalled
packages a couple of times because a crash would happen just after an
upgrade.

--
Loïc Minier

--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20100303124010.GB7516@bee.dooz.org

md5sums files

Submitted by Harald Braumann on Wed, 03/03/2010 - 07:13.

On Wed, Mar 03, 2010 at 03:06:20AM +0100, Wouter Verhelst wrote:
> In this day and age of completely and utterly broken MD5[0], I think we
> should stop providing these files, and maybe provide something else
> instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing
> md5sums.
>
> Or is it useful to be able to say "if it doesn't check out, it's
> certainly corrupt, and if it does check out, it may be corrupt"? Didn't
> think so.

As a means to check for filesystem corruptions or non-malicious changes,
MD5 is good enough. So until we have something better, I guess they can
stay.

But it would be great if the whole chain, from beginning to end, was
secured, even against a malicious and presumably very powerful attackers.
That would need:
* Package signatures
Currently only the release file is signed, but if you have a package
lying around, there is no way to check its authenticity.
* Cryptographically strong hashes for all files in the package
and a signature on the hash file.
Then you could really check the authenticity of all files on the system.
For the hash I would skip SHA-1 and move directly to SHA-256.

Oh, and a good read about the lifetime of hash algorithms can be found here: [0]

Cheers,
harry

[0] http://valerieaurora.org/hash.html

md5sums files

Submitted by Bernhard R. Link on Wed, 03/03/2010 - 07:51.

* Harald Braumann [100303 14:49]:
> As a means to check for filesystem corruptions or non-malicious changes,
> MD5 is good enough. So until we have something better, I guess they can
> stay.

I'd even say they should stay. The shorter the hash the more useable.
And md5 is the shortest well-defined and readily available everywhere.

> But it would be great if the whole chain, from beginning to end, was
> secured, even against a malicious and presumably very powerful attackers.

Checksums for files coming from packages is not really useful to defend
against attackers (it's really only reliablity and not security):

- an attacker can just divert any binary away and put it's own there.
- an attacker can just add some additional binary where it will override
another one (/sbin overriding /usr/sbin and so on).
- an attacker can add things to configuration and startup files
(thanks to .d directories you often not even need changing but only
adding files), including search binary or library paths, so one could
add binaries or behaviour changing libraries in directories not
looking that suspicious.

Most of those things can perhaps be fixed, but it needs much work
than just replacing some hash. (And many of those tasks might also
improve other areas (like http://packages.debian.org/cruft also having
the problem that packages create so many files and there is no way a
package can tell such programs where they are).

> For the hash I would skip SHA-1 and move directly to SHA-256.

And not to forget to add the size. Being allowed to create arbitrary
large files makes attacking interated hashes much easier...

Hochachtungsvoll,
Bernhard R. Link
--
"Never contain programs so few bugs, as when no debugging tools are available!"
Niklaus Wirth

md5sums files

Submitted by Goswin von Brederlow on Wed, 03/03/2010 - 04:14.

Mike Hommey writes:

> On Wed, Mar 03, 2010 at 08:29:09AM +0000, Neil Williams wrote:
>> On Wed, 3 Mar 2010 08:35:18 +0100
>> Mike Hommey wrote:
>>
>> > On Wed, Mar 03, 2010 at 06:30:34AM +0000, Sune Vuorela wrote:
>> > >
>> > > The md5 sums isn't to be used in case of a break in, as you can't trust
>> > > anything on the system anyways, but more things like:
>> > > - did I make; sudo make install something on top of packages
>> > > - did I just quickly hack a p{erl,ython}-script on the system to do
>> > > something different and forgot
>> >
>> > Which makes me think... wouldn't it be nice from dpkg to check the
>> > package files haven't been modified before upgrading ?
>>
>> No - if you're upgrading, you're doing it because you want to be sure
>> that the Debian version replaces the old version. However, if you do
>> use 'make; sudo make install; with a prefix of /usr instead
>> of /usr/local then a) you're taking a risk and b) md5sums are only a
>> partial protection. md5sums will NOT catch instances where the upstream
>> 'make install' provides files that are not part of the Debian package
>> nor will it catch files that have been moved as part of Debian
>> packaging. As these files are not put into place by dpkg anyway, then
>> they won't get cleaned up by dpkg and cannot be detected either by
>> using md5sums or by using dpkg. (dpkg will complain that certain
>> directories are not empty if the packaged files being upgraded are in
>> special places but not otherwise.) There's every chance that having
>> extraneous and/or duplicate content in the wrong places will break the
>> Debian package in ways that are not easily detectable and md5sums won't
>> help with that.
>>
>> Having md5sums around for simple checks like local admins copying
>> modified scripts over packaged versions is one thing but IMHO it does
>> not justify having md5sums in the wider case because a) local admin
>> changes are the responsibility of the local admin and b) md5sums only
>> help detect those changes where the admin changes a filename that
>> exactly matches the packaged name rather than adding more content /
>> cruft.
>>
>> 'sudo make install' into /usr is not something that md5sums can help us
>> fix and if that is the sole justification for retaining them then I
>> think that is a false argument.
>
> My point is that these people doing that, or even better, people doing
> an upgrade on a system where other people have been doing that would
> probably *want* to have a warning about the fact that the files dpkg is
> going to replace did *not* match what was supposed to be there in the
> first place.
>
> That would also possibly help spot filesystem errors, because when
> upgrading, dpkg is not going to write to the same places where the
> previous versions of the files were, and when it finishes upgrading, it
> will just remove the previous files and the corruptions, especially if
> they are due to hardware problems, will still be unnoticed and may
> affect more important data much later, when the freed space is used
> again.
>
> Mike

Run a nightly/weekly cron job that verifies all files. No point waiting
for a package upgrade to check this.

MfG
Goswin

md5sums files

Submitted by Wouter Verhelst on Wed, 03/03/2010 - 04:14.

On Wed, Mar 03, 2010 at 03:17:52PM +1100, Erik de Castro Lopo wrote:
> Russ Allbery wrote:
>
> > Wouter Verhelst writes:
> >
> > > Or is it useful to be able to say "if it doesn't check out, it's
> > > certainly corrupt, and if it does check out, it may be corrupt"? Didn't
> > > think so.
> >
> > I don't understand why you say this. Cryptographic attacks on MD5 aren't
> > going to happen as a result of random file corruption. The MD5 checksums
> > are still very effective at finding file corruption or modification from
> > what's in the Debian package unless that modification was done by a
> > sophisticated attacker (MD5 preimage attacks are still not exactly easy).
> > Detecting compromises is useful, but only a small part of what the MD5
> > checksums are useful for.
>
> If the machine has been compromised, *nothing* on the machine can be
> trusted, whether its gpg signed or not. However, for detecting corruptions
> and the local sysadmin meddling Russ mentioned, md5sum is more than adequate

Sure, I'm not contesting that.

> and using something 'more secure' than md5sum is overkill.

This is where I disagree. When a checksum algorithm is compromised (and
MD5 *is* compromised), things only ever get worse, not better. Indeed,
MD5 preimage attacks are pretty hard *today*. But switching to something
more secure in preparation for the day when MD5 will be easily cracked
by every script kiddo around is *not* overkill.

md5sums files

Submitted by Roland Mas on Wed, 03/03/2010 - 03:34.

Wouter Verhelst, 2010-03-03 03:06:20 +0100 :

[...]

> In this day and age of completely and utterly broken MD5[0], I think
> we should stop providing these files, and maybe provide something else
> instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop
> providing md5sums.
>
> Or is it useful to be able to say "if it doesn't check out, it's
> certainly corrupt, and if it does check out, it may be corrupt"?
> Didn't think so.

It is useful. Not too efficient against attacks, I'll grant you, but
there are other use cases. One of those I run into from time to time,
as maintainer of servers with webapps, is that I want to know from time
to time if there have been local changes in installed files when
compared to the (already custom) packages, so I can have a look and
integrate the changes into the next version of the custom packages. The
suggestion of having dpkg warn me on upgrade is interesting, but to be
proactive I'm happy to run debsums on my own before the deployment
stage. Because when you're in a deployment rush and one of the files
lacks a semicolon and breaks the whole app, you just $EDITOR
/usr/share/.../foo.php; some other times, your client messes with their
CSS files locally, and you don't want to be grumbled at if you lose that
change, even if you (and the client) know that the change *should* have
been done in the packages in the first place.

I'm quite okay with replacing or complementing the md5sums with
something stronger if it helps with security, but removing them
altogether… please no.

Roland.
--
Roland Mas

La menace de la baffe pèse plus lourd que la baffe elle-même.
-- in Sri Raoul le petit yogi (Gaudelette)

md5sums files

Submitted by Goswin von Brederlow on Wed, 03/03/2010 - 03:47.

Erik de Castro Lopo writes:

> Russ Allbery wrote:
>
>> Wouter Verhelst writes:
>>
>> > Or is it useful to be able to say "if it doesn't check out, it's
>> > certainly corrupt, and if it does check out, it may be corrupt"? Didn't
>> > think so.
>>
>> I don't understand why you say this. Cryptographic attacks on MD5 aren't
>> going to happen as a result of random file corruption. The MD5 checksums
>> are still very effective at finding file corruption or modification from
>> what's in the Debian package unless that modification was done by a
>> sophisticated attacker (MD5 preimage attacks are still not exactly easy).
>> Detecting compromises is useful, but only a small part of what the MD5
>> checksums are useful for.
>
> If the machine has been compromised, *nothing* on the machine can be
> trusted, whether its gpg signed or not. However, for detecting corruptions

Surely you are wrong. While you can not trust the gpg binary on the
compromised system the signatures will still hold. You just need to boot
from a live CD and use a garantied clean gpg and keyring to verify them.

> and the local sysadmin meddling Russ mentioned, md5sum is more than adequate
> and using something 'more secure' than md5sum is overkill.

Due to the lack of a signature on the md5sum file the file can not be
tusted fo security purposes at all. And for detecting unintentional
changes md5sum is plenty strong enough.

But imagine the file would be signed or a signature could be gotten
through a safe channel. Then the file could be used for a security audit
as well and something stronger would be benefitial. A simple way would
be create a shasum file (instead of md5) and to include a hash of said
file in Packages.gz.

> Erik

MfG
Goswin

md5sums files

Submitted by Russ Allbery on Tue, 03/02/2010 - 19:42.

Wouter Verhelst writes:

> Or is it useful to be able to say "if it doesn't check out, it's
> certainly corrupt, and if it does check out, it may be corrupt"? Didn't
> think so.

I don't understand why you say this. Cryptographic attacks on MD5 aren't
going to happen as a result of random file corruption. The MD5 checksums
are still very effective at finding file corruption or modification from
what's in the Debian package unless that modification was done by a
sophisticated attacker (MD5 preimage attacks are still not exactly easy).
Detecting compromises is useful, but only a small part of what the MD5
checksums are useful for. I'd more frequently use them to detect
well-intentioned but misguided meddling by a local sysadmin.

I certainly don't object to replacing them with SHA1 hashes, although
signed deb packages would still be my preferred solution to this problem.

--
Russ Allbery (rra@debian.org)

md5sums files

Submitted by Russ Allbery on Tue, 03/02/2010 - 23:44.

Erik de Castro Lopo writes:
> Russ Allbery wrote:

>> I don't understand why you say this. Cryptographic attacks on MD5
>> aren't going to happen as a result of random file corruption. The MD5
>> checksums are still very effective at finding file corruption or
>> modification from what's in the Debian package unless that modification
>> was done by a sophisticated attacker (MD5 preimage attacks are still
>> not exactly easy). Detecting compromises is useful, but only a small
>> part of what the MD5 checksums are useful for.

> If the machine has been compromised, *nothing* on the machine can be
> trusted, whether its gpg signed or not. However, for detecting
> corruptions and the local sysadmin meddling Russ mentioned, md5sum is
> more than adequate and using something 'more secure' than md5sum is
> overkill.

Also, while they're no substitute for a more thorough and careful check,
it's always worth remembering that most attackers are stupid or lazy.
Doing something simple frequently and something more thorough but
resource-intensive at a longer interval is often a good move. That said,
I would never recommend debsums as a security tool. It is, however,
useful when figuring out if anyone's been helpfully "fixing" things on a
system by editing scripts or replacing binaries that came in packages
originally.

--
Russ Allbery (rra@debian.org)

md5sums files

Submitted by Neil Williams on Wed, 03/03/2010 - 01:43.

On Wed, 3 Mar 2010 08:35:18 +0100
Mike Hommey wrote:

> On Wed, Mar 03, 2010 at 06:30:34AM +0000, Sune Vuorela wrote:
> >
> > The md5 sums isn't to be used in case of a break in, as you can't trust
> > anything on the system anyways, but more things like:
> > - did I make; sudo make install something on top of packages
> > - did I just quickly hack a p{erl,ython}-script on the system to do
> > something different and forgot
>
> Which makes me think... wouldn't it be nice from dpkg to check the
> package files haven't been modified before upgrading ?

No - if you're upgrading, you're doing it because you want to be sure
that the Debian version replaces the old version. However, if you do
use 'make; sudo make install; with a prefix of /usr instead
of /usr/local then a) you're taking a risk and b) md5sums are only a
partial protection. md5sums will NOT catch instances where the upstream
'make install' provides files that are not part of the Debian package
nor will it catch files that have been moved as part of Debian
packaging. As these files are not put into place by dpkg anyway, then
they won't get cleaned up by dpkg and cannot be detected either by
using md5sums or by using dpkg. (dpkg will complain that certain
directories are not empty if the packaged files being upgraded are in
special places but not otherwise.) There's every chance that having
extraneous and/or duplicate content in the wrong places will break the
Debian package in ways that are not easily detectable and md5sums won't
help with that.

Having md5sums around for simple checks like local admins copying
modified scripts over packaged versions is one thing but IMHO it does
not justify having md5sums in the wider case because a) local admin
changes are the responsibility of the local admin and b) md5sums only
help detect those changes where the admin changes a filename that
exactly matches the packaged name rather than adding more content /
cruft.

'sudo make install' into /usr is not something that md5sums can help us
fix and if that is the sole justification for retaining them then I
think that is a false argument.

Debian does move a lot of files around during packaging (the recent
stipulation against files in /var/www/ is just one instance) - my own
feeling is that the very packages that DO have their files moved around
for FHS reasons are the very ones that (some) local admins may want to
install from upstream tarballs ahead of the Debian package being
updated. Moving files invalidates any protection from having the
md5sum - unless the local admin retains a separate list of which files
have been installed separately, but then I thought the idea was that
such a record would be available by installing into /usr/local instead
of /usr in the first place. ;-)

Changing to SHA won't help. I'm for ditching all md5sums from packages.
It's not a lot of disc space gained but it does give a false sense of
security or 'insurance' if you want to avoid the more formal meaning of
'security'.

Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/

acetylcholine.com

Who's online

Navigation

User login

md5sums files

md5sums files

md5sums files

md5sums files

md5sums files

md5sums files

md5sums files

md5sums files

md5sums files

md5sums files

md5sums files

New forum topics

Recent comments