-d does the same as -b, except for actually building
anything! in effect, it does the same as -f (fetch)
except that the resulting variable assignments will
not be recursive (as with -f).
if -d is passed, configuration is still loaded, defconfig
files are still cycled through, and more importantly:
helper functions are still processed.
the grub, serprog and coreboot helper functions have
been modified to return early (zero status) if -d is
passed.
example usage:
./update trees -d coreboot x230_12mb
this would download the files, NOT build coreboot, and
NOT build the payloads.
there is one additional benefit to doing it this way:
the utils command has been removed, e.g.
./update trees -b coreboot utils default
the equivalent is now:
./update trees -d coreboot default
the overall effect of this change is that the trees script
no longer contains any project-specific logic, except for
the crossgcc build logic.
it does include some config/data mkhelper files at the top,
for serprog and coreboot, so that those variables defined in
those files can be global, but another solution to mitigate
that will also be implemented in a future commit.
the purpose of this and other revisions (in the final push
to complete lbmk audit 6 / cbmk audit 2) is to generalise as
much logic as possible, removing various ugly hacks.
Signed-off-by: Leah Rowe <leah@libreboot.org>
stub it from the trees script. the way it works now,
there is less code in the build system.
./build roms
this is no longer a thing
./build roms serprog
this is also no longer a thing. instead, do:
./update trees -b coreboot targetnamehere
./update trees -b pico-serprog
./update trees -b stm32-vserprog
the old commands still works, which causes the new
commands to run
coreboot roms now appear in elf/, not bin/, as before,
but those images now contain payloads.
NOTE: to contradict the above: ./build roms is no
longer a thing, in that it's now deprecated, but
backward compatibility is present for now. it will
be removed in a future release.
./build roms list also still works! it will do:
./update trees -b coreboot list
also:
./update trees -b grub list
this is now possible too
if a target "list" is provided, for multi-tree sources,
the targets are shown.
there is another difference: seagrub roms are now seagrub_,
instead of seabios_withgrub.
seabios-only roms are no longer provided, where grub is also
enabled; only seagrub is used. the user can easily remove
the bootorder file, if they want seabios to not try grub first.
Signed-off-by: Leah Rowe <leah@libreboot.org>
don't put it in the install modules.
this works around a hanging issue on haswell thinkpads.
when any usb device is inserted, GRUB will sometimes
hang if started from the SeaBIOS payload, *while* the
USB device is plugged in.
plugging in the USB device after GRUB starts worked.
it will have to be investigated more at a later date,
but this simply configuration change works.
the xhci module is already loaded explicitly, in grub.cfg
Signed-off-by: Leah Rowe <leah@libreboot.org>
stick the makeargs in mkhelper
i previously did cbmakeargs because the old revisions
had to define makeargs per-target otherwise. mkhelper
was done specifically to solve that problem.
Signed-off-by: Leah Rowe <leah@libreboot.org>
instead, make it a helper function, defined in target.cfg
this means that we can also do the same with other projects
in the future, and it is expected that we will have to.
these helper functions are used in cases where we want
additional actions to be performed.
actually, the helper could be anything. for example, you
could write:
mkhelper="./build foo bar"
and it would do that (at the point of execution, PWD
is the root directory of the build system)
Signed-off-by: Leah Rowe <leah@libreboot.org>
don't hardcode the check based on whether the current
project is grub. instead, define "btype" in target.cfg
if unset, we assume kconfig and permit kconfig commands
e.g. make menuconfig, make silentoldconfig, etc
this is to avoid the deadliest of sins:
project-specific hacks
Signed-off-by: Leah Rowe <leah@libreboot.org>
bios_extract, biosutilities and uefitool are used
by lbmk, but not by cbmk.
remove these. they were accidentally added during
cherry-picking of lbmk patches.
no harm done, because they are freue software packages
so they don't violate canoeboot's policy, but they are
nonetheless superfluous in the canoeboot project.
Signed-off-by: Leah Rowe <info@minifree.org>
Never, ever build images where GRUB is the primary payload.
These options have been removed from target.cfg handling:
* seabios_withgrub
* grub_withseabios
The "payload_grub" variable now does the same thing as
the old "seabios_withgrub" variable, if set.
The "grubonly" configuration is retained, and enabled by
default when SeaGRUB is enabled (non-grubonly also available).
Due to lbmk issue #216, it is no longer Libreboot policy to
make GRUB the primary payload on any board. GRUB's sheer size
and complexity, plus the large number of memory corruption issues
similar to it that *have* been fixed over the years, tells me
that GRUB is a liability when it is the primary payload.
SeaBIOS is a much safer payload to run as primary, on x86, due
to its smaller size and much more conservative development; it
is simply far less likely to break.
If GRUB breaks in the future, the user's machine is not
bricked. This is because SeaBIOS is the default payload.
Since I no longer wish to ever provide GRUB as a primary
payload, supporting it in lbmk adds needless bloat that
will later probably break anyway due to lack of testing,
so let's just assume SeaGRUB in all cases where the user
wants to use a GRUB payload.
You can mitigate potential security issues with SeaBIOS
by disabling option ROM execution, which can be done at
runtime by inserting integers into CBFS. The SeaBIOS
documentation says how to do this.
Libreboot's GRUB hardening guide still says how to add
a bootorder file in CBFS, making SeaBIOS only load GRUB
from CBFS, and nothing else. This, combined with the
disablement of option ROM execution (if using Intel
graphics), pretty much provides the same security benefits
as GRUB-as-primary, for example when setting a GRUB password
and GPG checks, with encrypted /boot as in the hardening guide.
Signed-off-by: Leah Rowe <leah@libreboot.org>
replace it with logic that simply uses "." to load
files directly.
config/git files are now directories, also containing
pkg.cfg files each with the same variables as before,
such as repository link and commit hash
this change results in a noticeable reduction in code
complexity within the build system.
unified reading of config files: new function setcfg()
added to lib.sh
setcfg checks if a config exists. if a 2nd argument is
passed, it is used as a return value for eval, otherwise
a string calling err is passed. setcfg output is passed
through eval, to set strings based on config; eval must
be used, so that the variables are set within the same
scope, otherwise they'd be set within setcfg which could
lead to some whacky results.
there's still a bit more more to do, but this single change
results in a substantial reduction in code complexity.
Signed-off-by: Leah Rowe <leah@libreboot.org>
this is bloat, because it's something the user can already
do at runtime configuration anyway.
set it to a reasonable default of 8 seconds instead of 5,
and don't honour the timeout variable in target.cfg.
this will be documented in the next release.
Signed-off-by: Leah Rowe <leah@libreboot.org>
the background is only a few kb. the whole rationale
before was to limit the space used in memdisk, but this
decision was made when the background was much bigger;
it has since been optimised greatly, and the grub modules
were heavily reduce, so it should be safe.
grub's memdisk breaks when you add too much data to it.
as part of simplifying the rest of lbmk, this change removes
some more bloat from the rest of lbmk. handling this in the
memdisk is much simpler than handling it with cbfstool.
Signed-off-by: Leah Rowe <leah@libreboot.org>
don't use the macports mirror, because it's not certain
whether those tarballs will always be there. use the
coreboot one as a backup instead, and nasm.us as main
Signed-off-by: Leah Rowe <leah@libreboot.org>
remove nvme support from the "default" grub tree
now there are three trees:
* default: no xhci or nvme patches
* nvme: contains nvme support
* xhci: contains xhci and nvme support
this is in case a bug like lbmk issue #216 ever occurs
again, as referenced before during lbmk audit 5
there is no indication that the nvme patch causes any
issues, but after previous experience i want to be sure
Signed-off-by: Leah Rowe <leah@libreboot.org>
i removed this before, when making grub multi-tree,
because the design i used in an earlier version of
the patch actually added the grub.elf generation
to grub source itself, but then i decided to hack
around the grub build system from lbmk/cbmk instead
re-add this functionality, so that users can easily
insert their own custom grub.cfg into cbfs without
needing to re-build their image.
Signed-off-by: Leah Rowe <leah@libreboot.org>
support redundant downloads, and enable inclusion of these
tarballs inside release archives, for offline builds.
Signed-off-by: Leah Rowe <leah@libreboot.org>
when we download coreboot, we currently don't have a way to
download crossgcc tarballs, so we rely on coreboot to do it,
which means running the coreboot build system to do it; which
means we don't get them in release archives, unless we add
very hacky logic (which did exist and was removed).
the problem with coreboot's build system is that it does not
define backup links for each given tarball, instead relying
on gnu.org exclusively, which seems OK at first because the
gnu.org links actually return an HTTP 302 response leading
to a random mirror, HOWEVER:
the gnu.org 302 redirect often fails, and the download fails,
causing an error. a mitigation for this has been to patch the
coreboot build system to download directly from a single mirror
that is reliable (in our case mirrorservice.org).
while this mitigation mostly works, it's not redundant; the
kent mirror is occasionally down too, and again we still have
the problem of not being able to cleanly provide crossgcc
tarballs inside release archives.
do it in config/submodules, like so:
module.list shall say the relative path of a given file,
once downloaded, relative to the given source tree.
module.cfg shall be re-used, in the same way as for git
submodules, but:
subfile="url"
subfile_bkup="backup url"
do this, instead of:
subrepo="url"
subrepo_bkup="backup url"
example entries in module.list:
util/crossgcc/tarballs/binutils-2.41.tar.xz
util/crossgcc/tarballs/gcc-13.2.0.tar.xz
util/crossgcc/tarballs/gmp-6.3.0.tar.xz
util/crossgcc/tarballs/mpc-1.3.1.tar.gz
util/crossgcc/tarballs/mpfr-4.2.1.tar.xz
util/crossgcc/tarballs/nasm-2.16.01.tar.bz2
util/crossgcc/tarballs/R06_28_23.tar.gz
the "subrev" variable (in module.cfg) has been renamed
to "subhash", so that this makes sense, and that name is
common to both subfile/subrepo.
the download logic from the vendor scripts has been re-used
for this purpose, and it verifies files using sha512sum.
therefore:
when specifying subrepo(git submodule), subhash will still
be a sha1 checksum, but:
when specifying subfile(file, e.g. tarball), subhash will
be a sha512 checksum
the logic for both (subrepo and subfile) is unified, and
has this rule:
subrepo* and subfile* must never *both* be declared.
the actual configuration of coreboot crossgcc tarballs
will be done in a follow-up commit. this commit simply
modifies the code to accomodate this.
over time, this feature could be used for many other files
within source trees, and could perhaps be expanded to allow
extracting source tarballs in leiu of git repositories, but
the latter is not yet required and thus not implemented.
Signed-off-by: Leah Rowe <leah@libreboot.org>
again: the "depend" variable must never be used for subprojects
that point to a subdirectory of the main project, because there's
no clean way of handling this in case of error conditions.
make it a submodule under config/submodules/. this is for the
documentation, including static site generator documentation,
and image files (photos).
as of this revision, there are now only those "depend" projects
defined in config/git/, where the destination directory of the
subject is not a subdirectory of the main project, so:
in a subsequest revision, i will mitigate an existing bug whereby
failure of the dependency project leaves the main one still
intact, breaking builds; this revision enables that to be done.
from now on, subproject-to-subdirectory-of-main-project will
be avoided in config/git/; config/submodules/ will be used.
Signed-off-by: Leah Rowe <leah@libreboot.org>
same as the previous patch, we must no longer use "define"
variables in config/git/ when the path is a subdirectory of
a given project, because it means that the download can only
happen after the main one, and currently if that fails, the
download of the main repo would remain intact, breaking future
builds in ways that we can't control - to be clear, it could
be controlled, but with added code complexity in the build
system, so:
put it in config/submodules/
Signed-off-by: Leah Rowe <leah@libreboot.org>
don't define it as a "depend" variable in config/git/,
because it means putting the files in a subdirectory of
an existing project was was already then downloaded, and
that means it can't be downloaded first; if the download
of it fails, the old download is left intact.
this bug isn't currently fixed in the build system, at all,
so this and other patches are being made to mitigate it.
Signed-off-by: Leah Rowe <leah@libreboot.org>
certain code checks for build.list, to skip it, for
example in items()
we already use config/data/grub to store grub config data
that applied to all trees
create these directories too:
config/data/coreboot
config/data/u-boot
config/data/seabios
move the respective build.list files in here, and also
to config/data/grub
now multi-tree projects contain, per directory, just the
target.cfg file and the patches directory. this is much
cleaner, because some of the logic can be simplified more
Signed-off-by: Leah Rowe <leah@libreboot.org>
it imports the same environmental variable fix because
i had the same buggy TMPDIR check there. i fixed that
upstream in untitled.
import the new untitled revision.
Signed-off-by: Leah Rowe <leah@libreboot.org>
The xHCI patches were removed because they caused issues
on Sandybridge-based Dell Latitude laptops. See:
https://codeberg.org/libreboot/lbmk/issues/216
The issue was not reported elsewhere, but we still don't
need xHCI support in Canoeboot's GRUB because none of the
available coreboot targets have xHCI support. However, we
may want it in the future and it helps to keep Canoeboot
in sync with Libreboot (this patch is adapted from lbmk).
Each given coreboot target can say which GRUB tree to use
by setting this in target.cfg:
grubtree="xhci"
In the above example, the "xhci" tree would be used. Some
generic GRUB config has been moved to config/data/grub/
and config/grub/ now looks like config/coreboot/ - also,
the grub.cfg file (named "payload" in each tree) is copied
to the GRUB source tree as ".config", then added to GRUB's
memdisk in the same way, as grub.cfg.
Several other design changes had to be made because of this:
* grub.cfg in memdisk no longer automatically jumps to one
in CBFS, but now shows a menuentry for it if available
* Certain commands in script/trees are disabled for GRUB,
such as *config make commands.
* gnulib is now defined in config/submodule/grub/, instead
of config/git/grub - and this mitigates an existing bug
where downloading gnulib first would make grub no longer
possible to download in lbmk.
There is another reason for merging this design change from
lbmk, and that reasoning also applies to lbmk. Specifically:
This change will enable per-board GRUB optimisation in the
future. For example, we hardcode what partitions and LVMs
GRUB scans because * is slow on ICH7-based machines, due
to GRUB's design. On other machines, * is reasonably fast,
for automatically enumerating the list of devices for boot.
Use of * (and other wildcards) could enable our GRUB payload
to automatically boot more distros, with minimal fuss. This
can be done at a later date, in subsequent revisions.
Signed-off-by: Leah Rowe <leah@libreboot.org>
it doesn't build, at present, but isn't used by any
coreboot targets, so the build issue does not come up
during release builds, but i did find it laying around
during my audits.
x86 qemu is on todo for libreboot, on all x86 boards,
but the current config is broken, so: remove it.
it's very much a requirement that anything in lbmk should
work.
Signed-off-by: Leah Rowe <leah@libreboot.org>
it's very unlikely that someone would use this
directory name nowadays, and i had half a mind
to remove it altogether
Signed-off-by: Leah Rowe <leah@libreboot.org>
in practise, the machines we support don't have
the option of including so many disks; 8 seems like
the most reasonable default. additionally, it's
unreasonable to expect *20 partitions*
this hardcoding is done to avoid using *, which is
slow in grub on some machines (the grub kernel always
re-enumerates the devices during every operation,
without caching any of it)
yet, the hardcoding is also slow; balance it a bit
better by searching fewer permutations, but not so few
that it would likely break a lot of setups
Signed-off-by: Leah Rowe <leah@libreboot.org>
we already supported syslinux but not grub
support grub by scanning for the most common paths,
based on the most popular distros
we don't hardcode this with * because it slows down
the boot, and in practise many distros still use the
same grub.cfg location as in BIOS systems (the EFI
one is often just a link to the BIOS one)
Signed-off-by: Leah Rowe <leah@libreboot.org>
this is a relic from the old days when we didn't
automated the grub.cfg logic as much. these days,
the grub.cfg logic is able to boot almost all distros
without any manual intervention or override.
removing these entries will speed up the boot in general
Signed-off-by: Leah Rowe <leah@libreboot.org>
the path "/boot/EFI" is unnecessary because the ESP
is always a FAT32 partition, so we don't need to
scan it as a subdirectory within a subdirectory.
the ESP is always mounted as its own partition,
FAT32, and EFI/ is always at the root of it
Signed-off-by: Leah Rowe <leah@libreboot.org>
the esp is always a fat32 partition so this makes no sensgrub.cfg: don't scan EFI on btrfs subvols
the esp is always a fat32 partition so this makes no sense
Signed-off-by: Leah Rowe <leah@libreboot.org>
replace variables ahcidev/atadev/nvmedev with a single
one named bootdev
the for loop goes through grub_scan_disk, so now it is
effectively a bootorder configuration
Signed-off-by: Leah Rowe <leah@libreboot.org>
it has always been gpl 3 or later, but it helps to have
the license declaration within the file
there's a copying file anyway. put spdx in the config
Signed-off-by: Leah Rowe <leah@libreboot.org>
Previously, grub_scan_disk could set ata, ahci or "both",
which would make both be tried (ahci first). This worked
when we only dealt with ata and ahci devices, but now we
support nvme devices so the logic is inherently flawed.
Instead, use grub_scan_disk to store the boot order, e.g.:
grub_scan_disk="ahci nvme ata"
grub_scan_disk="nvme ata"
In the first example, it would make GRUB scan ahci first,
then nvme and then ata.
In the secontd example, it would make GRUB scan nvme first,
and then ata.
If "both" is set, or anything other than ahci/ata/nvme,
grub_scan_disk is now changed to "nvme ahci ata".
Actual grub_scan_disk entries in target.cfg files will now
be modified, to match each machine.
Signed-off-by: Leah Rowe <leah@libreboot.org>
Fixes this bug:
https://codeberg.org/libreboot/lbmk/issues/216
Well, fix is the wrong word. We want xHCI ideally.
Mate is working on it as I write this. I've also:
* Disabled CONFIG_FINALIZE_USB_ROUTE_XHCI on Haswell
boards (coreboot)
* Disabled the GRUB payload on HP 820 G2 for now
We will need to re-add the xHCI patches once fixed.
If Mate/we can't fix it, I'll contact Patrick
Rudolph who originally wrote the xHCI patches.
Signed-off-by: Leah Rowe <leah@libreboot.org>
See:
https://codeberg.org/libreboot/lbmk/issues/216
Almost all users will be OK running GRUB, but a
minority of users have experienced a fatal error
pertaining to grub_free() or grub_realloc() (as
my investigation of GRUB sources reveal when grepping
the error reported in the link above).
We don't yet know what the bug is, only that the
error occurs, leading to an effective brick if the
user has GRUB as their primary payload.
So far, it has only been reported on some Intel
SandyBridge-based Dell Latitudes in Libreboot, but
we can't be too sure.
The user reported that memtest86+ passes just fine,
and SeaBIOS works; BIOS GRUB also works, which means
that the bug is likely only in an area of GRUB that
runs specifically on the coreboot payload, so it's
probably a driver in GRUB when running on the metal
rather than BIOS/UEFI.
The build system supports a configuration whereby
SeaBIOS is the primary payload, but GRUB is available
in the SeaBIOS boot select menu, and an additional
configuration is available where GRUB is what SeaBIOS
executes first (while still providing boot select);
both of these are now the *only* configurations
available, on all x86 targets except QEMU.
The QEMU target is fine because if the bug occurs there,
you can just close QEMU and try a different image.
Even after this bug is later identified and fixed,
the GRUB source code is vastly over-engineered and there
are likely many more such bugs. SeaBIOS is a reliable
payload; the code is small and robust. Remember always:
Code
equals
bugs
Therefore, this configuration change is likely going
to be permanent. This will apply in the next release.
Signed-off-by: Leah Rowe <leah@libreboot.org>
I've rebased the workaround-mx patch as follows. See:
commit 9a11cbf21a5078bcdb8db7584c44a9ee17020db4
Author: Nico Huber <nico.h@gmx.de>
Date: Fri Jan 13 01:19:07 2023 +0100
Let the flash context directly point to the used master
This change, now upstream in flashprog, made me have to do this in
the patch. I changed this:
flash->mst->spi.command(flash, sizeof(cmd), sizeof(buf), cmd, buf);
to this:
flash->mst.spi->command(flash, sizeof(cmd), sizeof(buf), cmd, buf);
It should work fine. This update imports the following upstream
patches from flashprog:
* 5b4fdd1 z60_flashprog.rules: Add udev rule for CH347
* 72c9e40 meson: Check for CPU families with known raw mem access
* 3458220 platform/meson: Port pciutils/pci.h workaround to Meson
* f279762 platform/meson: Check for libi386 on NetBSD
* 14da5f7 README: Convert to Markdown
* 8ddea57 README: Document branching and release policy
* 2522456 util/list_yet_unsupported_chips.sh: Fix path
* cbf9c11 spi: Don't cross 16MiB boundaries with long writes
* 823a704 dediprog: Skip warning on first attempt to read device string
* e8463c8 dediprog: Revise prefix check for given programmer id
* 38af1a1 dediprog: Revise id matching
* 4661e7c amd_spi100: Use flashprog_read_chunked() for progress reporting
* cdcfda2 read_memmapped: Use flashprog_read_chunked() for progress reporting
* 7679b5c spi25: Replace spi_read_chunked() with more abstract version
* ca1c7fd spi25: Normalize parameters of spi_nbyte_read()
* e36e3dc dediprog: Use default_spi_write_256
* 522a86d linux_spi: Use default_spi_read()/_write_256()
* 806509b cli_classic: Turn progress reporting into a progress bar
* 842d678 libflashrom: Return progress state to the library user
* aa714dd flashprog.c: Let select_erase_functions() return byte count
* 2eed4cf serprog: Add SPI Mode and CS Mode commands
* 821a085 dediprog: Implement id reading for SF600 and later
* 274e655 dediprog: Read device string early
* 0057822 dediprog: Add protocol detection for SF700 & SF600Plus-G2
* fb176d2 dediprog: Use more general 4BA write mode for newer protocols
* 0ab5c3d dediprog: Split device type and version parsing
* bdef5c2 dediprog: Use unsigned conversions to parse device string
* 5262e29 dediprog: Try to request 32B device string (instead of 16B)
* e76e21f dediprog: Get rid of some unnecessary hex constants
* 5a09d1e udelay: Lower the sleep vs delay threshold
* 03ad4a4 linux_mtd: Provide no-op delay implementation
* 211c6ec serprog: Refine flushing before synchronization
* 383b7fe serprog: Test synchronicity before trying to synchronize
* d7318ea serprog: Move synchronicity test into separate function
* 9a11cbf Let the flash context directly point to the used master
* aabb3e0 writeprotect: Hook wp functions into the chip driver
* 89569d6 memory_mapped: Reduce `decode_sizes` to a single `max_rom_decode`
* 929d2e1 internal: Pass programmer context down into chipset enables
* 7c717c3 internal: Pass programmer context down into board enables
* e3a2688 Pass programmer context to programmer->init()
* 2b66ad9 Start implementing struct flashprog_programmer
* 4517e92 memory_bus: Drop stale `size == 0` workaround and FIXME
* b197402 memory_bus: Split register mapping into own function
* 0e76d99 memory_bus: Move (un)map_flash_region into par master
* 9eec407 Perform default mapping only for respective chips
* 56b53dd wbsio_spi: Request memory mapping locally
* 5596190 it87spi: Request memory mapping locally
* 46449b4 spi25: Drop stale `bus == SPI` guards
* ab6b18f spi25: Move 4BA preparations into spi_prepare_4ba() hook
* 901fb95 Add prepare/finish_access() hooks for chip drivers
* a96aaa3 dediprog: Support long writes of 16MiB and more
* 1338936 Consider 4BA support when filtering erase functions
* 8d36db6 flashprog.8: Fix up serprog example
* d2ac303 flashprog.8: document new serprog cs parameter
* d1b9153 chipset_enable.c: Add Genoa to mendocino entry
Signed-off-by: Leah Rowe <leah@libreboot.org>
Leah Rowe