User:Allan/Pkgdiff
The Idea
Have a simple package "delta" system that allows downloading smaller amounts.
History
Previous implementations used xdelta3. This required doing the diff on uncompressed packages, and users needed to recreate the package on their own system (which was rather resource heavy). Switching to zstd compression in Arch packages was the beginning of the death of this approach, but a CVE in the implementation was the final nail.
New Idea
Create a package with just the files that have changed, and .MTREE, .PKGINFO, ...
A very quickly written example script to generate these package diffs (use at your own risk!)
Some Examples
bash
Update | Full Size | Diff Size | Percentage |
---|---|---|---|
5.0.018-1 to 5.0.018-2 | 1694723 | 448287 | 26% |
5.0.018-2 to 5.1.0-1 | 1732827 | 1671424 | 96% |
5.1.0-1 to 5.1.0-2 | 1732775 | 425654 | 25% |
5.1.0-2 to 5.1.004-1 | 1733041 | 429453 | 25% |
5.1.004-1 to 5.1.008-1 | 1734279 | 467479 | 27% |
5.1.008-1 to 5.1.012 | 1712112 | 471998 | 28% |
5.1.012-1 to 5.1.016 | 1707705 | 428130 | 25% |
coreutils
Update | Full Size | Diff Size | Percentage |
---|---|---|---|
8.32-1 to 9.0-1 | 2818996 | 2817404 | 100% |
9.0-1 to 9.0-2 | 2818602 | 48022 | 1.7% |
9.0-2 to 9.1-1 | 2760262 | 2761576 | 100% |
9.1-1 to 9.1-2 | 2777066 | 995875 | 36% |
9.1-2 to 9.1-3 | 2778174 | 20899 | 0.8% |
qt5-base
Header text | Header text | Header text | Header text |
---|---|---|---|
5.15.7+kde+r168-1 to 5.15.7+kde+r169-1 | 13658644 | 5601211 | 41% |
5.15.7+kde+r169-1 to 5.15.7+kde+r170-1 | 13654086 | 5291980 | 39% |
5.15.7+kde+r170-1 to 5.15.7+kde+r173-1 | 13658923 | 10140536 | 74% |
5.15.7+kde+r173-1 to 5.15.7+kde+r174-1 | 13660160 | 5620755 | 41% |
5.15.7+kde+r174-1 to 5.15.7+kde+r175-1 | 13660412 | 5626724 | 41% |
R
systemd
Update | Full Size | Diff Size | Percentage |
---|---|---|---|
251.7-2 to 251.7-3 | 7537985 | 4508871 | 60% |
251.7-3 to 251.7-4 | 7537942 | 4509277 | 60% |
251.7-4 to 252-1 | 7763035 | 7328268 | 94% |
252-1 to 252.1-1 | 7764080 | 4675494 | 60% |
252.1-1 to 252.1-2 | 7763575 | 4903232 | 63% |
Implementation Ideas
- Diff provided only between the current and previous package
- No support for diffs to earlier packages
- No diff chaining
- Only diffs below a threshold (80% ?) are provided
- Files in pkgdiff file get extracted like package files (including all . files in the root)
- .MTREE file is used to adjust times (& permissions?) on files that did not change in update
- Need to remove and files removed from package
TODO
- Investigate this approach on a much wider range of packages