don't do it after, because that means the main project
is saved under src/ before we know whether the subrepo
was downloaded.
the "depend" variable (in config/git/) is no longer used
for projects that go in subdirectories of a parent; now,
we use config/submodules/ for this type of dependency.
download the "depend" projects (as per config/git/) first.
this way, if they fail, the main one will fail, but if
they succeed and main fails, you can just run the main
download again and it won't fail.
this fixes a bug where, depending on how you download a
set of projects and depending on the order which you do so,
a given project can become un-downloadable on current design,
because git will complain that a directory already exists.
this fix is done not only in code (by this commit), but
by prior configuration changes.
Signed-off-by: Leah Rowe <leah@libreboot.org>
we're not checking for bad elfs, but the check itself was bad
due to a quirk in how sh works. really, really obscure bug.
fixed now!
if the given directory didn't actually exist, or there were no
files in it, it'd be searching for the file named "*"
which is obviously wrong
Signed-off-by: Leah Rowe <leah@libreboot.org>
don't check that the variable is empty
check that the file itself exists or not
this should fix the recent build issues
Signed-off-by: Leah Rowe <leah@libreboot.org>
in particular, the coreboot build system may auto-download
submodules when building cbfstool; vboot for instance.
we do not want such unpredictable behaviour, so now we
use UPDATED_SUBMODULES=1 when building coreboot utilities.
Signed-off-by: Leah Rowe <leah@libreboot.org>
one directory per util, under elf/
e.g. elf/cbfstool/
further split by tree name, e.g.:
elf/cbfstool/default/
elf/cbfstool/foo/
Signed-off-by: Leah Rowe <leah@libreboot.org>
this replicates the same behaviour as multi-tree builds,
checking for files inside the relevant elf/ directory
Signed-off-by: Leah Rowe <leah@libreboot.org>
the previous change makes memtest.bin get cached in elf/
but the path was being prefixed with src/ by script/roms
do away with the prefix
Signed-off-by: Leah Rowe <leah@libreboot.org>
it's also used from script/roms, in addition to trees
move these variables to a common file used everywhere
Signed-off-by: Leah Rowe <leah@libreboot.org>
certain code checks for build.list, to skip it, for
example in items()
we already use config/data/grub to store grub config data
that applied to all trees
create these directories too:
config/data/coreboot
config/data/u-boot
config/data/seabios
move the respective build.list files in here, and also
to config/data/grub
now multi-tree projects contain, per directory, just the
target.cfg file and the patches directory. this is much
cleaner, because some of the logic can be simplified more
Signed-off-by: Leah Rowe <leah@libreboot.org>
instead, check for the presence of target.cfg files
not in config/project/ but config/project/tree/
the way this check is done, it merely returns 1 if
config/project/*/target.cfg is detected, and returns
0 in all other cases, even if config/project/target.cfg
exists
that way, if the maintainer accidentally adds a
target.cfg in the main directory, the given multi-tree
project will not break
Signed-off-by: Leah Rowe <leah@libreboot.org>
adding help again is a bad idea. code should never
document itself; that's what documentation is for.
so, make the code do a better job telling the user
where to find documentation.
Signed-off-by: Leah Rowe <leah@libreboot.org>
The xHCI patches were removed because they caused issues
on Sandybridge-based Dell Latitude laptops. See:
https://codeberg.org/libreboot/lbmk/issues/216
The issue was not reported elsewhere, but we still don't
need xHCI support in Canoeboot's GRUB because none of the
available coreboot targets have xHCI support. However, we
may want it in the future and it helps to keep Canoeboot
in sync with Libreboot (this patch is adapted from lbmk).
Each given coreboot target can say which GRUB tree to use
by setting this in target.cfg:
grubtree="xhci"
In the above example, the "xhci" tree would be used. Some
generic GRUB config has been moved to config/data/grub/
and config/grub/ now looks like config/coreboot/ - also,
the grub.cfg file (named "payload" in each tree) is copied
to the GRUB source tree as ".config", then added to GRUB's
memdisk in the same way, as grub.cfg.
Several other design changes had to be made because of this:
* grub.cfg in memdisk no longer automatically jumps to one
in CBFS, but now shows a menuentry for it if available
* Certain commands in script/trees are disabled for GRUB,
such as *config make commands.
* gnulib is now defined in config/submodule/grub/, instead
of config/git/grub - and this mitigates an existing bug
where downloading gnulib first would make grub no longer
possible to download in lbmk.
There is another reason for merging this design change from
lbmk, and that reasoning also applies to lbmk. Specifically:
This change will enable per-board GRUB optimisation in the
future. For example, we hardcode what partitions and LVMs
GRUB scans because * is slow on ICH7-based machines, due
to GRUB's design. On other machines, * is reasonably fast,
for automatically enumerating the list of devices for boot.
Use of * (and other wildcards) could enable our GRUB payload
to automatically boot more distros, with minimal fuss. This
can be done at a later date, in subsequent revisions.
Signed-off-by: Leah Rowe <leah@libreboot.org>
this effectively lets you change the boot order. example:
./build roms -s "nvme ata" t1650_12mb
the above example would set:
grub_scan_disk="nvme ata"
another example:
./build roms -s nvme t1650_12mb
this would set:
grub_scan_disk="nvme"
this overrides what's set in target.cfg for the given
target. useful for quick reconfiguration if building
from source
Signed-off-by: Leah Rowe <leah@libreboot.org>
i already do this on crossgcc, but overlooked it on regular
builds where i just use -j, but coreboot's build system
makes use of the CPUS= option in make
use XBMK_THREADS for this
Signed-off-by: Leah Rowe <leah@libreboot.org>
Previously, grub_scan_disk could set ata, ahci or "both",
which would make both be tried (ahci first). This worked
when we only dealt with ata and ahci devices, but now we
support nvme devices so the logic is inherently flawed.
Instead, use grub_scan_disk to store the boot order, e.g.:
grub_scan_disk="ahci nvme ata"
grub_scan_disk="nvme ata"
In the first example, it would make GRUB scan ahci first,
then nvme and then ata.
In the secontd example, it would make GRUB scan nvme first,
and then ata.
If "both" is set, or anything other than ahci/ata/nvme,
grub_scan_disk is now changed to "nvme ahci ata".
Actual grub_scan_disk entries in target.cfg files will now
be modified, to match each machine.
Signed-off-by: Leah Rowe <leah@libreboot.org>
rather than if seabios_grubonly=y
if grubonly=y, still make the grubonly rom
this complements the previous commit
Signed-off-by: Leah Rowe <leah@libreboot.org>
as stated in the previous commit, i'm adding this function
to lbmk because there are files i want to systematically
delete in libreboot releases, not just canoeboot releases,
but libreboot releases delete things such as unlicensed
readme files, or poorly licensed other files.
i initially moved the nuke function to its own file so as
to reduce the number of merge conflicts when merging
changes to git.sh between cbmk and lbmk, but if they're
going to both contain this file, then it makes sense
to have this in git.sh once again.
Signed-off-by: Leah Rowe <info@minifree.org>
blobs.list is now nuke.list
this is because i'm going to import this functionality
into lbmk (libreboot build system).
libreboot will not do full deblobbing like canoeboot does,
but there are still certain files that i like to delete
in releases, such as u-boot's strlcat.c file under tests
calling it "nukeblobs" in libreboot makes no sense, but
i like to avoid merge conflicts when cherry-picking
patches between cbmk and lbmk, so i like to make sure
that functions and variables common to both are named
the name.
simply calling it "nuke" or calling the files "nuke.list"
is probably inoffensive while conveying the same meaning.
Signed-off-by: Leah Rowe <info@minifree.org>
the inject script is from lbmk and referenced here, but
cbmk does not have this script.
remove the reference.
Signed-off-by: Leah Rowe <info@minifree.org>
it wasn't being reset before. when coreboot is being
built, i add to makeargs every time. if multiple targets
are being built, the make command would end up looking
something like:
make -C src/coreboot/default UPDATED_SUBMODULES=1 \
UPDATED_SUBMODULES=1
(the parameter would be printed twice)
of course, this doesn't check whether that parameter is
added already in target.cfg for a given target, but that's
ok because i won't add that one in target.cfg
i baked it into the code, only when handling coreboot,
because that was easier than either putting it in makeargs
for every coreboot target.cfg, or again modifying the code to
handle that; the current solution is the cleanest.
Signed-off-by: Leah Rowe <leah@libreboot.org>
we do not want submodules to be downloaded after the fact.
we only handle this on ./update trees -f coreboot
Signed-off-by: Leah Rowe <leah@libreboot.org>
i'm importing some changes from lbmk and they go at the
end of git.sh, in the diffs. moving the deblob function
to its own file will allow me to cherry-pick with fewer
merge conflicts.
Signed-off-by: Leah Rowe <info@minifree.org>
firstly, memtest86+ is currently not cross compiled and
relies on 64-bit headers (x86_64 only). a 32-bit distro
is unlikely to be able to build 64-bit binaries.
secondly: vboot throws a build error due to -Werror when
building on 32-bit hosts. we rely on vboot code to build
cbfstool, so turn off -Werror on vboot
that's all. 32-bit hosts are not recommended; it is assumed
that you are building on an x86_64 host. work will go into
the build system at a later date to make it more portable,
by cross compiling everything, but this should fix 32-bit
for now.
there are some x60/t60 users who still want to build roms,
so let's allow them that possibility.
Signed-off-by: Leah Rowe <leah@libreboot.org>
an equivalent change has been made in lbmk.
certain cbmk-specific variable names have been made
generic, with certain functions and other variables
moved around.
i maintain sync between libreboot and canoeboot, where
both projects can have the same behaviours, and most of
the merge conflicts have to do with variable names
containing "LBMK", "lbmk", "cbmk" or "CBMK", or
indeed "canoeboot" and "libreboot"
LBMK/lbmk/CBMK/cbmk variables between canoeboot and
libreboot now contain the string XBMK/xbmk
it should now be *much* easier to merge build system
changes between lbmk and cbmk.
Signed-off-by: Leah Rowe <info@minifree.org>
i always say, code should never document itself.
that's what documentation is for. the releases
contain documentation under docs/ but the git
repository does not; for that, use the website.
(in practise, cbmk usually needs internet anyway)
Signed-off-by: Leah Rowe <leah@libreboot.org>
in cbmk, we call check_project() to set variables
such as projectname, version, version date
this is unnecessary, because all main scripts use
this functionality anyway
do it by default
Signed-off-by: Leah Rowe <leah@libreboot.org>
export CBMK_RELEASE="y"
if this is done, the tarball is created instead
of a directory, and the rom images are nuked using
./vendor inject with the nuke option, inserting the
correct version files; the rom directory is deleted
now the release script logic simple renames existing
tarballs. the benefit of this change is fewer lines of
code, and now cbmk doesn't use an insane amount of disk
space when building a *lot* of release images (the
uncompressed directories are deleted after each build)
Signed-off-by: Leah Rowe <leah@libreboot.org>
the release variable is all we need, turning a target on
or off for a given release.
the status checks were prone to bugs, and unnecessary; it
also broke certain benchmark scripts.
it's better to keep the cbmk logic simpler. board status
will be moved to the documentation instead.
Signed-off-by: Leah Rowe <leah@libreboot.org>
there are only two scripts under script/ now, and there
probably won't be many more. cbmk's design has simplified
to such a degree that the two-level directory structure is
no longer necessary.
the existing command structure has not changed. for example:
./build roms list
./update trees -f coreboot default
these will still work, but the symlinks to "build" are now
strictly for backwards compatibility; they may be removed
at a later date, but i'll keep the current design for now.
this also leads to a quirk, for example:
./build roms all
./update roms all
these now do the exact same thing, whereas "./update roms all"
would have previously been an invalid command.
Signed-off-by: Leah Rowe <leah@libreboot.org>
the main script isn't that big, and since the main
purpose of cbmk is geared toward the releases, it
makes sense to reduce the number of scripts by
merging into the main one
the way this works, "./update release" still works
afterward
so, the way cbmk is used shall remain unchanged
Signed-off-by: Leah Rowe <leah@libreboot.org>
previous command:
./build serprog
now it is:
./build roms serprog
after that, it's the same arguments e.g.
./build roms serprog stm32
./build roms serprog rp2040
further cleanup to commence
Signed-off-by: Leah Rowe <leah@libreboot.org>
x is part of the for loop in main() and may or not
still be available from handle_target, depending on
your implementation of sh, but this should not be assumed
do it properly
Signed-off-by: Leah Rowe <leah@libreboot.org>
for example:
./build roms list stable
this lists all images that are marked "stable"
now:
./build roms list _stable
this lists all images that are *not* marked stable
this will help me keep track during development
Signed-off-by: Leah Rowe <leah@libreboot.org>
This is useful on desktops, where you want GRUB to
automatically start, but you still want access to the
GRUB menu, in the case where you rely on SeaBIOS to
execute the VGA ROM inside your graphics card.
Signed-off-by: Leah Rowe <leah@libreboot.org>
just to ensure that nothing goes wrong. we don't rely on
the status variable for releases, because there is another
variable, release, that target.cfg files declare, e.g.
release="n"
release="y"
you can just omit the variable, because it defaults to y, so
you only need declare it when it needs to be "n"
Signed-off-by: Leah Rowe <leah@libreboot.org>
export CBMK_STATUS=n
if not set, the status checks and confirmation dialogs
persist. if set to y they persist.
if you set it to n, all checks are disabled, so e.g.:
./build roms all
this would once again build all targets, regardless
of status. this is if you want the old behaviour.
Signed-off-by: Leah Rowe <leah@libreboot.org>
for example:
./build roms list
this will list every now, still. same behaviour. now see:
./build roms list stable
this will list all stable roms
./build roms list untested
this lists untested roms. but wait!
./build roms list untested broken unstable
./build roms list broken unstable
yes. it works this way. now you can use lbmk to easily
see what rom status are, during maintenance.
Signed-off-by: Leah Rowe <leah@libreboot.org>
export CBMK_VERSION_TYPE=x
x can be: stable, unstable
in target.cfg files, specify:
status=x
x can be: stable, unstable, broken, untested
if unset, cbmk defaults to "unknown"
if CBMK_VERSION_TYPE is set, no confirmation is asked
if the given target matches what's set (but what's set
in that environmental variable can only be stable or
unstable)
if CBMK_RELEASE="y", no confirmation is asked, unless
the target is something other than stable/unstable
"unstable" means it works, but has a few non-breaking
bugs, e.g. broken s3 on dell e6400
whereas, if raminit regularly fails or it is so absolutely
unreliable as to be unusable, then the board should be
declared "broken"
untested means: it has not been tested
With this change, it should now be easier to track whether
a given board is tested, in preparation for releases. When
working on trees/boards, status can be set for targets.
Also: in the board directory, you can add a "warn.txt" file
which will display a message. For example, if a board has a
particular quirk to watch out for, write that there. The message
will be printed during the build process, to stdout.
If status is anything *other* than stable, or it is unstable
but CBMK_VERSION_TYPE is not set to "unstable", and not building
a release, a confirmation is passed.
If the board is not specified as stable or unstable, during
a release build, the build is skipped and the ROM is not
provided in that release; this is in *addition* to
release="n" or release="y" that can be set in target.cfg,
which will skip the release build for that target if "n"
Signed-off-by: Leah Rowe <info@minifree.org>
the temporary rom per build was not being deleted after
finishing the current target. this adds up in /tmp during
large builds, when building for many targets. fix this!
Signed-off-by: Leah Rowe <leah@libreboot.org>
release="n" can be set in target.cfg on coreboot targets
script/update/release exports CBMK_RELEASE="y"
script/build/roms skips building a given target if release="n"
in target.cfg *and* CBMK_RELEASE="y"
you could also do the export yourself before running ./build roms,
for example:
export CBMK_RELEASE="y"
./build roms all
this will be used in subsequent revisions, to exclude certain
targets from the next canoeboot release.
Signed-off-by: Leah Rowe <info@minifree.org>
cbmk otherwise uses nproc to set the number of build threads,
in these places:
* generic make commands in script/update/trees
* crossgcc make command in script/update/trees
the -T0 option is also used in script/update/release, when running
tar.
with this change, you can do:
export CBMK_THREADS=x
where x is the number of threads. when you then run
cbmk, your chosen number of threads will override
the default. this may be useful on a host that does
not have a lot of memory.
Signed-off-by: Leah Rowe <info@minifree.org>
in shell scripts, a function named the same as a program included in
the $PATH will override that program. for example, you could make a
function called ls() and this would override the standand "ls".
in cbmk, a part of it was first trying to run the "fail" command,
deferring to "err", because some scripts call fail() which does
some minor cleanup before calling err.
in most cases, fail() is not defined, and it's possible that the user
could have a program called "fail" in their $PATH, the behaviour of
which we could not determine, and it could have disastrous effects.
cbmk error handling has been re-engineered in such a way that the
err function is defined in a variable, which defaults to err_ which
calls err_, so defined under include/err.sh.
in functions that require cleanup prior to error handling, a fail()
function is still defined, and err is overridden, thus:
err="fail"
this change has made xx_() obsolete, so now only x_ is used. the x_
function is a wrapper that can be used to run a command and exit with
non-zero status (from cbmk) if the command fails. the xx_ command
did the same thing, but called fail() which would have called err();
now everything is $err
example:
rm -f "$filename" || err "could not delete file"
this would now be:
rm -f "$filename" || $err "could not delete file"
overriding of err= must be done *after* including err.sh. for
example:
err="fail"
. "include/err.sh"
^ this is wrong. instead, one must do:
. "include/err.sh"
err="fail"
this is because err is set as a global variable under err.sh
the new error handling is much cleaner, and safer. it also reduces
the chance of mistakes such as: calling err when you meant to
call fail. this is because the standard way is now to call $err,
so you set err="fail" at the top of the script and all is well.
Signed-off-by: Leah Rowe <info@minifree.org>
./update release -m u-boot
if someone just wants to make u-boot, they can
use this and it tars up all the trees.
Signed-off-by: Leah Rowe <info@minifree.org>
in some cases, the build system was needlessly, and sometimes
erroneously, creating crossgcc symlinks, which then caused an
issue, namely:
in lbmk release builds, dell e6400 is build before fam15h boards,
and it sets xtree, but fam15h_rdimm doesn't, and later this would
cause fam15h_rdimm boards to use xtree="default" (because they don't
set xtree), causing the newer toolchain to be used on coreboot 4.11.
this patch fixes the issue. quite a simple problem, actually.
Signed-off-by: Leah Rowe <leah@libreboot.org>
the changelog file is only present in releases, so
use the presence of this file for the test.
someone who wants to fetch projects within a release
archive can simply use the git repo, or delete the file.
Signed-off-by: Leah Rowe <leah@libreboot.org>
use the git log, as follows:
git log --graph --pretty=format:'%Cred%h%Creset %s %Creset' --abbrev-commit
this creates a nice, uniform list of changes.
Signed-off-by: Leah Rowe <leah@libreboot.org>
let them specific it, rather than falling back
to coreboot/default (can also be used for coreboot boards)
Signed-off-by: Leah Rowe <leah@libreboot.org>
There is no need to add multiple keymap files, because
GRUB can load keymaps from CBFS. The current build logic
is designed to avoid building multiple GRUB binaries,
which are expensive computationally because each one
would then have to be compressed for each board.
This patch provides the best of both worlds: less space
used in flash like in the old lbmk design (1 keymap per
board), but retaining the current build speeds and therefore
not re-introducing the slowness of lbmk's previous GRUB
build logic.
The grub.cfg file has been modified, accordingly. It now
only loads a keymap.gkb file from CBFS, by default. It does
this, only if that file exists; if not, GRUB already defaults
to US Qwerty layout anyway.
ALSO: compress all keymap gkb files with xz -6
GRUB automatically decompresses files when accessed.
This results in about 2KB of flash space saved in CBFS.
Here is real-world data, showing the increased flash space:
< fallback/payload 0x3eb80 simple elf 548821 none
< keymap.cfg 0xc4bc0 raw 16 none
< (empty) 0xc4c00 null 11633316 none
---
> fallback/payload 0x3eb80 simple elf 546787 none
> keymap.gkb 0xc43c0 raw 344 none
> (empty) 0xc4540 null 11635044 none
This was taken by diffing the cbfstool "print" output,
both before and after. The *after* result is with this change.
11633316. In this example, 1728 bytes have been saved. Therefore,
with compression taken into account, this patch saves about 1.7KB
of space in CBFS.
This change means that lbmk can now scale to support hundreds
of keymaps, without increasing the amount of flash space used,
in each given image. Since the keymap files are compressed in
lbmk.git, in advance, we spend no additional time on compression
at build time. The resulting change in build speed in negligible.
Adding your own keymap.gkb file was already possible, for changing
the keymap in libreboot images, if you didn't want to change the
memdisk (and thus re-compile grub.elf). Now, this is the default
behaviour, and the only way to do it. It's much more efficient.
The original keymap files can be restored, by running unxz.
Signed-off-by: Leah Rowe <leah@libreboot.org>
the "kmapdir" variable was removed in an earlier audit,
but was overlooked for -k because that option was untested.
rather than initialise the variable, re-use grubcfgsdir.
this fix enables e.g. "-k usdvorak" to work again.
Signed-off-by: Leah Rowe <leah@libreboot.org>
cbmk 9429287 is the present canoeboot revision, on this day,
two commits after canoeboot 20231107
the cbmk revision was based on lbmk c4d90087, but lbmk
has developed a lot since, right up to f5b04fa5. lbmk
c4d90087 was four commits after libreboot 20231106
this patch brings cbmk up to date, versus lbmk f5b04fa5,
which is 135 commits after libreboot 20231106 (not 4)
therefore, the next canoeboot release shall import lbmk
changes made *after* lbmk revision f5b04fa5. good day!
In English (the above is for my reference, next time
I make a new canoeboot release):
This imports all of the numerous improvements from
Libreboot, sans the non-FSDG-compliant changes. You
can find a full list of such changes in the audit4 page:
https://libreboot.org/news/audit4.html
A full canoeboot-ised changelog will be available in
the next canoeboot release, with these and subsequent
changes. Most notable here is the update to the new
GRUB 2.12 release (instead of 2.12-rc1), and the
improvements Riku made to pico-serprog. And the build
system improvements from lbmk, such as improved, more
generic cmake and autoconf handling.
Canoeboot-specific changes: I also tweaked the deblob
logic, to make it less error-prone. The new design
changes imported into cbmk (based on latest lbmk) somewhat
broke the deblob logic; it was constantly reminding the
user that blobs.list was missing for coreboot,
at config/coreboot/blobs.list - coreboot is a multi-tree
project in both cbmk and lbmk, and the deblob logic was
tuned for single/multi, but was treating coreboot as both.
for simplicity, i removed the check for whether blobs.list
is present. this means that the operator must ensure that
these files are present, in any given revision, where they
are required on a given set of projects (and the files are
all present, in this update to cbmk)
Also of note: the grub.cfg improvements are included in this
cbmk update. The improved grub.cfg can find grub/syslinux
configs by default, not just grub anymore, also finds extlinux,
and will also find them on EFI System Partition - in addition,
UEFI-based install media is also more robust; although cbmk
doesn't provide UEFI configurations on x86, our GRUB palyoad
does still need to work with distro install media, and many
of them now use UEFI-based GRUB configurations in their
installation media, which just happen to work with our GRUB
Signed-off-by: Leah Rowe <leah@libreboot.org>
changes after libreboot 20231101 were imported,
up to libreboot 20231106, and then to revision:
c4d90087535617d4fb31ca94803f9426010cfec5
i945 and gm45 configs were re-done, and dell e6400
was moved to its own tree with the ddr2 fix moved
there, to prevent breakage on ddr3-based gm45 boards
(look at libreboot 20231106 for more info)
several fixes are present in this canoeboot release,
that were only merged in libreboot *after* the
libreboot 20231106 release, and they are:
* c4d90087 add grub mods: diskfilter,hashsum,loadenv,setjmp
* d0d6decb re-add grub modules: f2fs, json, read, scsi, sleep
* 86608721 nvmutil: print usage
* f12f5c3a nvmutil: fix makefile
the release documentation has also been updated,
pulling down newer cbwww and cbwww-img based on
the new canoeboot 20231107 release announcement
Signed-off-by: Leah Rowe <leah@libreboot.org>
Leah Rowe