The build system already deletes .git in all source
directories for each given release, but does so at
the very end; it still does, but now it is deleted
one by one per project, to save space during very
large builds (release sizes vary wildly, depending
on how many trees exist for coreboot basically).
If you're building entirely in tmpfs (as I do), this
could be a problem if you have lots of .git/ directories.
This change reduces disk usage, or in the above example,
memory usage when running the build system from tmpfs.
This complements another recent change, where ROM images
are compressed per target during release builds, rather
than all at the very end of the process. It is part of a
series of optimisations, to reduce the memory and disk
usage of the build system, and to reduce I/O wastage
in general.
This change will not be the last of such changes!
Signed-off-by: Leah Rowe <leah@libreboot.org>
an equivalent change has been made in cbmk.
certain lbmk-specific variable names have been made
generic, with certain functions and other variables
moved around.
i maintain sync between libreboot and canoeboot, where
both projects can have the same behaviours, and most of
the merge conflicts have to do with variable names
containing "LBMK", "lbmk", "cbmk" or "CBMK", or
indeed "canoeboot" and "libreboot"
LBMK/lbmk/CBMK/cbmk variables between canoeboot and
libreboot now contain the string XBMK/xbmk
it should now be *much* easier to merge build system
changes between lbmk and cbmk.
Signed-off-by: Leah Rowe <leah@libreboot.org>
it's only used from main() in the main build script,
and it's very small, as is main()
therefore, move the logic into main()
Signed-off-by: Leah Rowe <leah@libreboot.org>
in lbmk, we call check_project() to set variables
such as projectname, version, version date
this is unnecessary, because all main scripts use
this functionality anyway
do it by default
Signed-off-by: Leah Rowe <leah@libreboot.org>
releases aren't reproducible anyway. we were
using options available in gnu tar for this.
it will be revisited at a later date. however, the next
time this is done, we will use another method because
there are in fact portable ways to create tarballs
reproducibly, documented on reproducible-builds.org
to be revisited, at a later date. for now, remove bloat.
Signed-off-by: Leah Rowe <leah@libreboot.org>
it's only called once, from this file, within a small
function, and the function itself is very small.
remove, and put the contents of the function in the
calling function.
Signed-off-by: Leah Rowe <leah@libreboot.org>
export LBMK_RELEASE="y"
if this is done, the tarball is created instead
of a directory, and the rom images are nuked using
./vendor inject with the nuke option, inserting the
correct version files; the rom directory is deleted
now the release script logic simple renames existing
tarballs. the benefit of this change is fewer lines of
code, and now lbmk doesn't use an insane amount of disk
space when building a *lot* of release images (the
uncompressed directories are deleted after each build)
Signed-off-by: Leah Rowe <leah@libreboot.org>
the release variable is all we need, turning a target on
or off for a given release.
the status checks were prone to bugs, and unnecessary; it
also broke certain benchmark scripts.
it's better to keep the lbmk logic simpler. board status
will be moved to the documentation instead.
Signed-off-by: Leah Rowe <leah@libreboot.org>
export LBMK_THREADS=x
where x is an integer. this is already supported for
setting the number of build threads, but if not set
it uses nproc.
openbsd doesn't have nproc. default to 1 thread.
now you MUST set threads. e.g. in linux do:
export LBMK_THREADS=$(nproc)
preliminary work is being done to make lbmk run
on openbsd!
Signed-off-by: Leah Rowe <leah@libreboot.org>
lbmk otherwise uses nproc to set the number of build threads,
in these places:
* generic make commands in script/update/trees
* crossgcc make command in script/update/trees
the -T0 option is also used in script/update/release, when running
tar.
with this change, you can do:
export LBMK_THREADS=x
where x is the number of threads. when you then run
lbmk, your chosen number of threads will override
the default. this may be useful on a host that does
not have a lot of memory.
Signed-off-by: Leah Rowe <leah@libreboot.org>
in shell scripts, a function named the same as a program included in
the $PATH will override that program. for example, you could make a
function called ls() and this would override the standand "ls".
in lbmk, a part of it was first trying to run the "fail" command,
deferring to "err", because some scripts call fail() which does
some minor cleanup before calling err.
in most cases, fail() is not defined, and it's possible that the user
could have a program called "fail" in their $PATH, the behaviour of
which we could not determine, and it could have disastrous effects.
lbmk error handling has been re-engineered in such a way that the
err function is defined in a variable, which defaults to err_ which
calls err_, so defined under include/err.sh.
in functions that require cleanup prior to error handling, a fail()
function is still defined, and err is overridden, thus:
err="fail"
this change has made xx_() obsolete, so now only x_ is used. the x_
function is a wrapper that can be used to run a command and exit with
non-zero status (from lbmk) if the command fails. the xx_ command
did the same thing, but called fail() which would have called err();
now everything is $err
example:
rm -f "$filename" || err "could not delete file"
this would now be:
rm -f "$filename" || $err "could not delete file"
overriding of err= must be done *after* including err.sh. for
example:
err="fail"
. "include/err.sh"
^ this is wrong. instead, one must do:
. "include/err.sh"
err="fail"
this is because err is set as a global variable under err.sh
the new error handling is much cleaner, and safer. it also reduces
the chance of mistakes such as: calling err when you meant to
call fail. this is because the standard way is now to call $err,
so you set err="fail" at the top of the script and all is well.
Signed-off-by: Leah Rowe <leah@libreboot.org>
This is of Broadwell platform, one generation above Haswell.
Of note: this uses HP Sure Start. Although the flash is 16MB,
our CBFS section (and IFD configuration) assumes 12MB flash,
so the final 4MB will be left unflashed on installation,
after blanking the private flash. The coreboot documents have
more information about this.
Some minor design changes in lbmk were made, to accomodate
this port:
Support for extracting refcode binaries added (pulled from
Google recovery images). The refcode file is an ELF that
initialises the MRC and the PCH. It is also responsible for
enabling or disabling the Intel GbE device, where Google
does not enable it, but lbmk modifies it per the instructions
on the coreboot documentation, so as to enable Intel GbE.
Google's recovery image stores the refcode as a stage file,
but coreboot changed the format (for CBFS files) after 4.13
so coreboot 4.13's cbfstool is used to extract refcode. This
realisation made me also change the script logic to use a
cbfstool and ifdtool version matching the coreboot tree, for
all parts of lbmk, whereas lbmk previously used only the
default tree for cbfstool/ifdtool, on insertion and deletion
of vendor files - it was 81dc20e744 that broke extraction of
refcode on google's recovery images, where google used an older
version of cbfstool to insert the files in their coreboot ROMs.
A further backported patch has been added, copying coreboot
revision f22f408956 which is a build fix from Nico Huber.
Iru Cai submitted an ACPI bugfix after the revision lbmk
currently uses, for coreboot/default, and this fix is
needed for rebooting to work on Linux 6.1 or higher. This
patch has been backported to lbmk, while it still uses the
same October 2023 revision of coreboot.
Broadwell MRC is inserted at the same offset as Haswell,
so I didn't need to tweak that.
Signed-off-by: Leah Rowe <leah@libreboot.org>
the script can now also handle autoconf build systems,
whereas this could previously only be done for grub.
with this change, the overall sloccount is also lower
Signed-off-by: Leah Rowe <leah@libreboot.org>
while seemingly pedantic, this does actually make code
easier to read. mostly just switching to shorthand for
variable names, where no expansions or patterns are used
Signed-off-by: Leah Rowe <leah@libreboot.org>
this part of the code *must* return. the for loop
afterwards must not be permitted to execute.
it's unlikely that this would ever occur, unless
perhaps the user is using a very buggy sh.
Signed-off-by: Leah Rowe <leah@libreboot.org>
it wasn't being copied right
the roms under elf/ were being copied, but not the ones
under bin/ - i need to audit it further
for now, i run modify_coreboot_roms from build/roms
instead of update/trees
so, the ones under elf/ no longer have bootblocks copied.
it's only done in bin/
Signed-off-by: Leah Rowe <leah@libreboot.org>
keymaps weren't being set in keymay.cfg of cbfs, due
to use of x_ in the rom script, and x_ doesn't handle
quotes or spaces in arguments well.
i'm going to remove use of x_ and xx_ (it's in my todo),
for next release.
for now, hot patch the release. i've gone through and
replaced use of x_ with || err, in some places.
not just the keymap.cfg command, but others too. in case
there are more issues we missed.
this commit is being tagged "20231021fix" and i'm using
this tag to re-build the 20231021 release. i'll just
replace the tarballs in rsync and add errata to the news
page announcing the release. all i did was break peoples
umlauts, i didn't brick their machines fortunately!
very minor bug. anyway, x_/xx_ is a great idea, but sh
isn't really designed for that style of programming. i'll
go back to using just || err in the next release.
Signed-off-by: Leah Rowe <leah@libreboot.org>
in some cases, use of x_ or xx_ can be error-prone,
due to the way $@ is handled; commands requiring
quotes, or with funny file names as arguments such
as spaces in the file name, or other special
characters, can make the x/xx functions break.
in those cases, where x/xx must not be used, the
commands use || err instead
in other cases, use of x/xx is superfluous, and has
been removed in some commands.
Signed-off-by: Leah Rowe <leah@libreboot.org>
as opposed to the current 3-level structure.
recent build system simplifications have enabled
this change, thus:
./build fw coreboot -> ./build roms
./build fw grub -> ./build grub
./build fw serprog -> ./build serprog
./update project release -> ./update release
./update project trees -> ./update trees
./update vendor download -> ./vendor download
./update vendor inject -> ./vendor inject
alper criticised that the commands were too long,
so i made them shorter!
Signed-off-by: Leah Rowe <leah@libreboot.org>
move it all to other files where items are used, and not
used anywhere else. this reduces the size of vendor.sh.
also remove a few redundant variables, or variables that
are not meaningfully used.
a few items have been moved to include/option.sh
Signed-off-by: Leah Rowe <leah@libreboot.org>
Just one script.
Just one!
Well, two, but the 2nd one already existed:
logic in update/project/trees and
update/project/repo was merged into
include/git.sh and update/project/build
was renamed to update/project/trees; an -f
option was added, which calls the functions
under git.sh
so git clones are now handled by the main build
script (for handling makefiles and defconfigs)
but the logic there is a stub, where git.sh
does all the actual heavy lifting
this cuts the file count down by two, and reduces
sloccount a reasonable amount because much of
the logic already exists in the build script, when
it comes to handling targets. git.sh was adjusted
to integrate with this, rather than act standalone
Signed-off-by: Leah Rowe <leah@libreboot.org>
errors are not defined for mktemp, and the /tmp file
system should be assumed reliable.
if /tmp is *unreliable*, then this is not something that
lbmk either can or should fix; the user clearly has
bigger problems.
manpages for mktemp do not define errors. it is assumed
to be completely reliable.
Signed-off-by: Leah Rowe <leah@libreboot.org>
Instead of having detailed error messages, run most
commands through a function that calls err() under
fault conditions.
Where detail is still required, err() is still called
manually. Where it isn't, the error message is simply
whatever command was executed to cause the error.
This results in a massive sloccount reduction for lbmk;
specifically, 178 sloc reduction, or a 8.1% reduction.
The total sloccount is now 2022, for shell scripts.
Signed-off-by: Leah Rowe <leah@libreboot.org>
update/blobs/download and update/project/repo both use
the same logic, for setting variables with awk and a
specially formatted configuration file.
unify this logic under include/option.sh, and use that.
Signed-off-by: Leah Rowe <leah@libreboot.org>
Leah Rowe