From 75cf273e58a7bed7eea86559b1adffe45515c2da Mon Sep 17 00:00:00 2001 From: Leah Rowe Date: Mon, 11 Sep 2023 15:35:28 +0100 Subject: [PATCH] Libreboot Build System Audit 2 Signed-off-by: Leah Rowe --- site/news/MANIFEST | 1 + site/news/audit2.md | 332 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 333 insertions(+) create mode 100644 site/news/audit2.md diff --git a/site/news/MANIFEST b/site/news/MANIFEST index 5b47741..b4f2542 100644 --- a/site/news/MANIFEST +++ b/site/news/MANIFEST @@ -1,3 +1,4 @@ +audit2.md argon2.md hp8470p_and_dell_t1650.md hp2170p.md diff --git a/site/news/audit2.md b/site/news/audit2.md new file mode 100644 index 0000000..2687a5a --- /dev/null +++ b/site/news/audit2.md @@ -0,0 +1,332 @@ +% Libreboot Build System Audit 2 (new Libreboot release soon!) +% Leah Rowe +% 11 September 2023 + +This is *not* a new release announcement, but a new release *is* coming soon! +This new audit has been the opening act, now concluded and the priority will +now shift once again to new hardware support. + +Introduction +============ + +A lot of work has gone into Libreboot since the [20230625 +release](libreboot20230625.md), the main focus of which has been yet another +audit; the [previous audit](audit.md) was mild by comparison. + +Great care is always taken when writing for [lbmk](../docs/maintain/), which +is Libreboot's build system, but in the past, auditing was always done ad-hoc. +Since the start of 2023, auditing is now a main aspect of Libreboot development, +and the result has been that the *quality* of Libreboot's build system has +improved greatly. The same care has been given to documentation aswell. + +Many bugs have been fixed, and the Libreboot build system (lbmk) is generally +much more efficient now, compared to the Libreboot 20230625 release. The main +focus has been this audit, now concluded, and the next focus shall once again +be adding more mainboards to Libreboot, with a view to making a full new release +some time during September 2023. + +Before diving into details, here is a brief summary of the recent audit, and +the changes made under it, which accounts for almost *200* individual changes. +This is the *summarised* list, and may not list all changes, but it's pretty +exhaustive: + +* Very general auditing has been done, finding and fixing bugs. +* Reduced the number of scripts significantly. There were about 50 scripts in + the Libreboot 20230625 build system. There are closer to *20* in today's + Libreboot revision. +* *Massively reduced the size of the build system*: where only shell scripts are + concerned, the total sloccount in Libreboot 20230625 was 3388 source lines. + As of *today*, that figure stands at 2644 source codes, a *22% reduction in + code size*. +* Many scripts that were separate are now unified. For example: the scripts + handling defconfigs files on SeaBIOS, u-Boot and coreboot have now been + merged into a single script, performing the same work *better* in less code. +* Ditto many other scripts; repeated logic unified, logic generalised. The + logic for *downloading* coreboot and u-boot was unified into one script, + basing off of the coreboot one, and then expanding to also cover SeaBIOS. + Most building (e.g. handling of Makefiles) is now done in a single script. +* Far superior error handling; in many scripts, the `-e` option in `sh` was + heavily relied upon to catch errors, but now errors are handled much more + verbosely. *Many* fault conditions previously did not make lbmk *exit* at all, + let alone with non-zero status, and zero status was sometimes being returned + under some edge cases that were tested. Error handling is more robust now. +* `util/ich9utils` (containing `ich9gen`) was *removed*, thus eliminating about + 3000 source lines (of C code) from lbmk. The `nvmutil` program, also provided + by and originating from the Libreboot project, can already change GbE MAC + addresses. Coreboot's bincfg can generate ich9m descriptors, and ifdtool can + manipulate them; so the features provided by ich9utils were superfluous, since + they are available in other projects that we ship. We now ship pre-built + ifd/gbe configs on these machines, which can be modified or re-assembled + manually if you want to. This eliminates a moving part from Libreboot, and + speeds up the build a little bit. +* ROM images (of coreboot) build *much faster*: no-payload coreboot ROMs are + cached on disk, as are payloads, where previously only the latter was cached. + These cached images have as much inserted into them as possible, to eliminate + redundant steps in the build process. The `elf` directory contains these, and + the existing `bin` directory still holds the full ROM images (containing + payloads) when compiled. +* GRUB payload: vastly reduced the size of the payload, by eliminating GRUB + modules that were not needed. About 100KB of compressed space saved in flash! +* GRUB payload: [argon2 key derivation supported](argon2.md) - this means LUKS2 + decryption is now possible in GRUB. This work was performed by Nicholas + Johnson, rebasing from Axel's AUR patch for GRUB 2.06 (Libreboot currently + uses GRUB 2.12). +* Blobutil: generally more reliable now at downloading vendor files, especially + under fault conditions; for example, if a download failed before, it'd try + a backup link, but now it also tries the backup link if main download succeeds + but checksum verification didn't; and SHA512 checksums are now used, for + greater security, whereas Libreboot 20230625 used sha1sum (now we use + sha512sum). A user agent is specified in wegt, matching that used by Tor + Browser (which in turn mimics Firefox running on Windows). This is needed + for some vendors, which seem to dislike wget's default user agent. +* The *new* coding style is now used on many more scripts, including + the `build/boot/roms_helper` script - the new style is much cleaner, + mandating that logic be top-down, with a `main()` function defined; it's + basically inspired by the OpenBSD coding style for C programs, adapted to + shell scripts. +* All GRUB keymaps now included; a single `grub.elf` is now used on all ROM + images. The `grub.cfg` goes in GRUB memdisk now, but can be overridden by + inserting a `grub.cfg` in CBFS; many behaviours are also controlled this way, + for example to change keymaps and other behaviours. This results in *much* + faster builds, because a different GRUB payload doesn't have to be added to + each new ROM image; such takes time, due to time-expensive LZMA compression. + This, plus the optimised set of GRUB modules, also makes GRUB itself load + much faster. All of the fat has been trimmed, though still quite a lot more + than a Crumb. +* A lot of scripts have been removed entirely, and their logic not replaced; + in many cases, Libreboot's build system contained logic that had gone unused + for many years. +* More reliable configs now used on desktop mainboards: SeaBIOS-only for start, + but GRUB still available where feasible (in the SeaBIOS menu). This makes it + more fool proof for a user who might use integrated graphics and then switch + to a graphics card; the very same images will work. +* TMPDIR environmental variable now set, and exported from main parent process + when running lbmk; child processes inherit it, and a single tmp dir is used. + This is then automatically cleaned, upon exit from lbmk; previously, lbmk did + not cleanly handle `/tmp` at all, but now it's pretty reliable. + +Several other features have been added too, despite the overall code size +reduction; for example, Riku Viitanen implemented support for building RP2040 +images e.g. Pico Pi, and stm32-vserprog images, for use in small MCU-based SPI +flashers (via serprog protocol). + +Literally about 200+ changes have been made to the Libreboot build system, + +FULL list of changes (from lbmk git log) +--------------------------------------- + +Almost all of the changes are post-20230625 release, and almost all of them +are audit-related, so I've simply pasted every commit between the last release +and now. A few of them, e.g. the new mainboard ports, are not audited-related. + +Entries that give time frames such as *"14 hours ago"* are relative to +today, 11 September 2023, at some time around 3PM UK time. Changes on top are +newer, and changes further down are older: + +``` +* c400916e - (HEAD -> master) coreboot/hp8200sff_4mb: fix bad ifd path in config (14 hours ago) +* 087f0e06 - make lbmk help text actually vaguely helpful (19 hours ago) +* 093d40ee - build/release/src: be more thorough deleting .git (21 hours ago) +* 630a6546 - build/release/src: delete elf/ in srcdir (21 hours ago) +* 0543350d - handle/make/file: run make-clean first (21 hours ago) +* 12f9afe6 - build/release/src: remove cbutils/ in srcdir (21 hours ago) +* fe00ab4e - build/release/src: remove errant code (21 hours ago) +* be4ed540 - handle/make/config: distclean once per tree (21 hours ago) +* f227cc08 - handle/make/config: fix distclean/crossgcc-clean (22 hours ago) +* 669c9770 - handle/make/config: fix whitespace and 80-line bug (23 hours ago) +* d28ad6aa - build/release/roms: use -T0 on serprog tarballs (23 hours ago) +* 308c21dd - build/boot/roms stragglers: properly handle errors (23 hours ago) +* c16b28ef - build/release/src: re-create symlinks, don't copy (2 days ago) +* 32dcf9e5 - coreboot/qemu_x86_12mb: re-add this mainboard (2 days ago) +* 5aef8156 - scripts: use printf, not echo, where appropriate (2 days ago) +* 76e12cd4 - update/blobs printf statements: use double quotes (2 days ago) +* 84bf47b5 - scripts: better handling of printf: stdout/stderr (2 days ago) +* b78009e2 - checkgit: properly print output to stderr (2 days ago) +* f45f5e62 - update/project/*: remove redundant checks (2 days ago) +* 3e76e70d - blobs/download: don't use the -B option in make (2 days ago) +* 877c691e - build/release/roms: remove errant line break (2 days ago) +* f03efbc2 - blobs/inject: add error condition on rm command (2 days ago) +* 20be007f - blobs/inject: fix checksum validation if no-ucode (2 days ago) +* f989d5b4 - blobs/sources: fix backup links on some files (2 days ago) +* 878550d5 - use sha512sum to check downloads, not sha1sum (2 days ago) +* 022e0200 - Merge pull request 'Add stm32-vserprog' (#129) from Riku_V/lbmk:stm32 into master (2 days ago) +|\ +| * bed444ff - Add stm32-vserprog (2 days ago) +* | e9e4ada5 - build/boot/rom: only insert scan.cfg if needed (3 days ago) +* | 0e3f3efc - build/boot/roms: delete tmpcfg when done (3 days ago) +* | a69e8548 - set grub.cfg timeout to 5s (10 on some boards) (3 days ago) +* | 4a459b02 - Merge pull request 'pico-serprog improvements' (#128) from Riku_V/lbmk:master into master (3 days ago) +|\| +| * 7b6fb958 - Build pico-serprog binary release archive (3 days ago) +| * c292e01b - Build for all pico board, not just the "original" (3 days ago) +| * 1bde6bb3 - Support multiple dependencies per project (3 days ago) +| * 4d3b16da - Cleaner parent directory creation (4 days ago) +|/ +* 7e8465be - grub: re-add module: play (5 days ago) +* e3b9dfc9 - util/nvmutil: put code all in nvmutil.c (6 days ago) +* 8fc5f6ed - update/blobs/inject: use tmp/romdir, not TMPDIR (7 days ago) +* da991262 - build/release/roms: use tmp/romdir, not TMPDIR (7 days ago) +* 15081ed9 - grub: make backgrounds configurable in target.cfg (7 days ago) +* 0d315c3a - curl/wget downloads: set re-try count to 3 (7 days ago) +* bdf171e3 - don't use /tmp/ for git clones. use tmp/ instead. (7 days ago) +* 196f293a - build/release/roms: fix ucode handling (7 days ago) +* c0c7f3ae - build/release/roms: simplify defcongic handling (7 days ago) +* a56cad71 - update/blobs: unify global variables (7 days ago) +* 2cbc7eea - update/blobs/*: unify checking of defconfig files (7 days ago) +* 52677309 - update/blobs/extract: replace errant target code (7 days ago) +* ea7fae97 - build/boot/roms: don't create empty bin/ directory (7 days ago) +* c62a4239 - update/blobs/inject: remove errant target handling (7 days ago) +* 950166da - update/blobs/download: remove errant code (7 days ago) +* 0668d234 - add checkversion to build/release/src (7 days ago) +* c92a596c - grub: remove xnu module (7 days ago) +* e659ddd8 - grub: remove legacy file system modules (7 days ago) +* cf535785 - re-add grub modules cat, eval and pbkdf2 (7 days ago) +* 33e6088a - move script/misc/versioncheck to main directory (7 days ago) +* 2c769dc1 - move me7_update_parser.py to util/ (7 days ago) +* da3c9bb3 - merge config/ and resources/ (8 days ago) +* a0501050 - blobs/download: don't handle ifd/gbe files (8 days ago) +* 03788d14 - move ifd/gbe configs into config/ifd/ (8 days ago) +* 6ddb0e09 - run make oldconfig on coreboot/default mainboards (8 days ago) +* 19efdf9e - ich9m mainboards: use pre-assembled ifd/gbe files (8 days ago) +* af8d8cda - add ich9m ifd/gbe files (8 days ago) +* d554efae - build/release/src: copy e6430 ifd/gbe (8 days ago) +* 09aae7be - build/rpi-pico-serprog: better error handling (8 days ago) +* 1dc54608 - fix rpi-pico builds when running it twice (8 days ago) +* c63052cf - fix memtest86plus download/build (8 days ago) +* fb4e6834 - Merge pull request 'Add Dell Latitude E6430' (#124) from nic3-14159/lbmk:e6430 into master (8 days ago) +|\ +| * ebc04e52 - Add Dell Latitude E6430 (8 days ago) +* | 71d361aa - Merge pull request 'Less cat abuse' (#123) from Riku_V/lbmk:cat into master (8 days ago) +|\ \ +| * | ef3fb05d - Less cat abuse (8 days ago) +* | | eebf7133 - switch repo links for pico-serprog (8 days ago) +| |/ +|/| +* | 9ef8a7ea - Merge pull request 'Automate pico-serprog builds' (#122) from Riku_V/lbmk:mkserprog into master (9 days ago) +|\| +| * e369e8fb - automate rpi-pico serprog builds (9 days ago) +|/ +* 92b4db69 - build/release/src: only clean kbc1126 if it exists (9 days ago) +* 7c6b35cf - unify build/clean scripts: use handle/make instead (9 days ago) +* cec37747 - build/release/*: use -T0 in xz, for multithreading (9 days ago) +* b4b63adb - don't support ucode removal on untested targets (9 days ago) +* b30c7e33 - coreboot/e6400: support nvidia models (9 days ago) +* 436b2ccb - handle/make/config -m/-u: actually copy configs (9 days ago) +* 3c7b09ac - handle/make/config: properly handle cbutils (9 days ago) +* a3bc7ccd - handle/make/file: fix uefitool builds (9 days ago) +* 4885c796 - handle TMPDIR from include/export.sh (9 days ago) +* 56f16bc8 - don't do cmake on uefitool if the Makefile exists (10 days ago) +* 98d1ea5a - build/release/src: bugfix: actually copy cb/ub/sb (10 days ago) +* 755f925a - build/release/src: copy handle symlink (10 days ago) +* 3ad29d2d - build/release/src: remove Makefile reference (10 days ago) +* d69c231e - build/release/src: fix bad variable reference (10 days ago) +* 38440153 - update build/release/src based on lbmk changes (10 days ago) +* 0e782e7e - update the fetch scripts themselves (10 days ago) +* 98f30b6d - build/coreboot/utils: exit 1 if target.cfg missing (10 days ago) +* b9662fbe - handle project downloads in main lbmk script (10 days ago) +* 12b33eb8 - lbmk script: always clean up /tmp files (11 days ago) +* 225e2609 - only remove microcode in build/release/roms (11 days ago) +* bf774acf - move build/boot/rom moverom to handle/make/config (11 days ago) +* e5546128 - build/release/roms: fix syntax error (11 days ago) +* fbda0f04 - re-add /dev/null redirect on . ${1} (11 days ago) +* b2bad5a0 - build/release/src: copy the include/ directory (13 days ago) +* eb54e427 - grub: all one grub.elf containing keymaps and cfg (2 weeks ago) +* c6fd4d2a - lbmk: run ./build dependencies *before* root check (2 weeks ago) +* 6722624d - build/boot/roms: fix bad variable assignment (2 weeks ago) +* 55be6dda - dependencies/ubuntu2004: update based on debian (2 weeks ago) +* 0052f9d0 - fix: don't require git config for dependencies (2 weeks ago) +* 6dbddf85 - build/boot/roms: simplify ich9m ifd handling (2 weeks ago) +* f5787c9e - build/boot/roms ich9m ifd: use fast dd command (2 weeks ago) +* d9292cec - build/boot/roms: use the new coding style (2 weeks ago) +* 4623f3f2 - Remove superfluous GRUB modules (save CBFS space) (2 weeks ago) +* 623c3389 - fix typo in error message ("as not permitted") (2 weeks ago) +* 4a280c62 - .gitcheck: re-write entirely. force global config. (2 weeks ago) +* 355eb765 - move resources/scripts/ to script/ (2 weeks ago) +* eed34d3e - enable memtest86plus on various boards (2 weeks ago) +* bc0fb51d - x86 desktops: only enable seabios_withgrub (2 weeks ago) +* 9457d6be - unified list command for all scripts (2 weeks ago) +* 93d2dcad - handle/make/config: add missing pipes for err (2 weeks ago) +* 0e6851c8 - delete the Makefile (2 weeks ago) +* ebbefa60 - handle/config/file: rename to handle/make/config (2 weeks ago) +* df6db1c6 - handle/config: fix errant "handle src for" call (2 weeks ago) +* 6874bc39 - "handle src for" - change to handle make file (2 weeks ago) +* 798ce03a - handle/config: add missing error handle (2 weeks ago) +* 29a8193e - build/src/for: rename to handle/make/file (2 weeks ago) +* 27c67295 - handle/config/file: unified distclean handling (2 weeks ago) +* 197464bc - build/src/for: use -j for multithreaded builds (2 weeks ago) +* 95f290d9 - build/release/src: update based on recent changes (2 weeks ago) +* 5a47c01b - scripts: put quotes around file/directory names (2 weeks ago) +* 1c8401be - much, much stricter, more verbose error handling (2 weeks ago) +* 50c395df - .gitcheck: continue if no .git (don't break) (3 weeks ago) +* be7a5b0c - .gitcheck: must stricter error handling (3 weeks ago) +* 3a5ba57f - .gitcheck: only redirect stdout to /dev/null (3 weeks ago) +* 8f4f0e00 - use the new coding style in scripts (3 weeks ago) +* 4c6c7d10 - scripts: never exit 1, always call err instead (3 weeks ago) +* 52f3fd35 - blobs/download: copy dl_path, don't move it (3 weeks ago) +* 57adbc6e - unify err functions across scripts (3 weeks ago) +* b3fbcdf6 - .gitignore: ignore *all* seen files (3 weeks ago) +* 24f09335 - Merge pull request 'hp8300usdt: enable mSATA' (#118) from Riku_V/lbmk:master into master (3 weeks ago) +|\ +| * df1e8913 - hp8300usdt: enable mSATA (3 weeks ago) +* | dfb93166 - Merge pull request 'memtest86+ v6.20' (#116) from Riku_V/lbmk:memtest into master (3 weeks ago) +|\| +| * fa926632 - memtest86+ v6.20 (3 weeks ago) +|/ +* 1bd84209 - Merge pull request 'osbmk->lbmk' (#117) from Riku_V/lbmk:osbmk-lbmk into master (3 weeks ago) +|\ +| * 7be203dd - osbmk->lbmk (3 weeks ago) +* | 04ee2672 - also clean up the main scripts (3 weeks ago) +* | 62f23123 - general code cleanup on lbmk shell scripts (3 weeks ago) +|/ +* 7be47065 - unify build/defconfig and modify/defconfig (3 weeks ago) +* 0faf2a0c - main lbmk script: exit non-zero if argc is wrong (3 weeks ago) +* 6e92d9a3 - fix "./build help" (3 weeks ago) +* 9031bb7b - unify dependencies scripts (3 weeks ago) +* 023d6b69 - unify build/clean into ./build release src (3 weeks ago) +* f893a29b - unify most module build scripts (3 weeks ago) +* c83d1a8d - unify grub scripts under one directory (3 weeks ago) +* 438bf2c9 - grub/modules.list: add argon2 (3 weeks ago) +* fd602532 - grub: import phc argon2 implementation (for luks2) (3 weeks ago) +* 2c0c521e - bump grub revision a bit (3 weeks ago) +* e076d893 - unify update/modify coreboot/u-boot/seabios script (3 weeks ago) +* e25984d7 - remove board: qemu_x86_12mb (not usable for now) (3 weeks ago) +* e5b898f6 - consolidate u-boot/seabios/coreboot build scripts (3 weeks ago) +* 673b144a - coreboot/fam15h: fix for gcc/gnat building (4 weeks ago) +* 63b0e99f - don't call blobutil directly from lbmk (4 weeks ago) +* 08486227 - remove download scripts, consolidate into script (4 weeks ago) +* 8459e33b - improve user feedback in blobutil (4 weeks ago) +* 59dba6cf - merge coreboot/u-boot download logic to one script (4 weeks ago) +* 2453c303 - gitclone: always clean up /tmp (4 weeks ago) +* adeb065c - fix permissions on arch dependencies script (4 weeks ago) +* 6075fed8 - NEW BOARD: HP EliteBook 8470p (Intel GPU) (4 weeks ago) +* f9afeb6f - NEW BOARD: Dell Precision T1650 (4 weeks ago) +* f8f77cb2 - NEW BOARD: HP EliteBook 2170p (5 weeks ago) +* c5c89467 - Merge pull request 'Update 'README.md'' (#89) from ewpr5kwu/lbmk:master into master (5 weeks ago) +|\ +| * 5204f0a9 - Update 'README.md' (2 months ago) +* cb8bf380 - bump seabios revision to 30 May 2023 (5 weeks ago) +* 27ee975e - bump grub revision to 2.12-rc1 (5 weeks ago) +* 705149a3 - coreboot/default: bump revision to 2 August 2023 (5 weeks ago) +* 22ee7f74 - blobs/download: save ME file to correct location (5 weeks ago) +* cdd83ab1 - blobs/download: try backup if bad hash on main (5 weeks ago) +* f18b1859 - blobs/download: support more formats on ME extract (5 weeks ago) +* f0efaf79 - add unar to dependencies scripts (5 weeks ago) +* e8ba0f87 - blobs/download: declare full user agent (6 weeks ago) +* 4875eef1 - blobs/download: properly handle backup/main url (6 weeks ago) +* cca93ca3 - blobs/download: don't download backup on main (6 weeks ago) +* 3aeefaa7 - blobs/download: set common user agent string (6 weeks ago) +* 5e83d2bc - blobs/download: simplify for loop (6 weeks ago) +* 8f1d3ad1 - scripts: fix indentation in switch/case blocks (6 weeks ago) +* 748e0972 - blobutil/ec: abort if kbc1126 ec extraction fails (7 weeks ago) +* e594ac16 - coreboot/fam15h: remove unused files (7 weeks ago) +* 44bd077a - Revert "build/boot/roms mkUBootRoms: initialise variables" (8 weeks ago) +* 7c90a407 - build/boot/roms mkUBootRoms: initialise variables (8 weeks ago) +* d918139f - coreboot/fam15h: re-enable microcode updates (8 weeks ago) +* 8c777428 - board/qemu_x86: don't enable u-boot (8 weeks ago) +* fb44c349 - coreboot/haswell: fix acpica downloads (8 weeks ago) +* af084014 - coreboot: re-add asus kgpe-d16/kcma-d8/kfsn4-dre (8 weeks ago) +* e6002b91 - coreboot/cros: fix acpica downloads (8 weeks ago) +* f34e07ae - build/boot/roms: fix coreboot-version in releases (9 weeks ago) +``` + +A new Libreboot release is planned soon.