User:Allan/Pkgdiff

From ArchWiki

The Idea

Have a simple package "delta" system that allows downloading smaller amounts.

History

Previous implementations used xdelta3. This required doing the diff on uncompressed packages, and users needed to recreate the package on their own system (which was rather resource heavy). Switching to zstd compression in Arch packages was the beginning of the death of this approach, but a CVE in the implementation was the final nail.

New Idea

Create a package with just the files that have changed, and .MTREE, .PKGINFO, ...

A very quickly written example script to generate these package diffs (use at your own risk!)

Some Examples

bash

Update Full Size Diff Size Percentage
5.0.018-1 to 5.0.018-2 1694723 448287 26%
5.0.018-2 to 5.1.0-1 1732827 1671424 96%
5.1.0-1 to 5.1.0-2 1732775 425654 25%
5.1.0-2 to 5.1.004-1 1733041 429453 25%
5.1.004-1 to 5.1.008-1 1734279 467479 27%
5.1.008-1 to 5.1.012 1712112 471998 28%
5.1.012-1 to 5.1.016 1707705 428130 25%

coreutils

Update Full Size Diff Size Percentage
8.32-1 to 9.0-1 2818996 2817404 100%
9.0-1 to 9.0-2 2818602 48022 1.7%
9.0-2 to 9.1-1 2760262 2761576 100%
9.1-1 to 9.1-2 2777066 995875 36%
9.1-2 to 9.1-3 2778174 20899 0.8%

qt5-base

Caption text
Header text Header text Header text Header text
5.15.7+kde+r168-1 to 5.15.7+kde+r169-1 13658644 5601211 41%
5.15.7+kde+r169-1 to 5.15.7+kde+r170-1 13654086 5291980 39%
5.15.7+kde+r170-1 to 5.15.7+kde+r173-1 13658923 10140536 74%
5.15.7+kde+r173-1 to 5.15.7+kde+r174-1 13660160 5620755 41%
5.15.7+kde+r174-1 to 5.15.7+kde+r175-1 13660412 5626724 41%

R

systemd

Caption text
Update Full Size Diff Size Percentage
251.7-2 to 251.7-3 7537985 4508871 60%
251.7-3 to 251.7-4 7537942 4509277 60%
251.7-4 to 252-1 7763035 7328268 94%
252-1 to 252.1-1 7764080 4675494 60%
252.1-1 to 252.1-2 7763575 4903232 63%

Implementation Ideas

  • Diff provided only between the current and previous package
    • No support for diffs to earlier packages
    • No diff chaining
    • Only diffs below a threshold (80% ?) are provided
  • Files in pkgdiff file get extracted like package files (including all . files in the root)
  • .MTREE file is used to adjust times (& permissions?) on files that did not change in update
  • Need to remove and files removed from package

TODO

  • Investigate this approach on a much wider range of packages