Commit Graph

335 Commits (ef867ecccad5464690c4fa342946fa2c7a4eac3d)

Author SHA1 Message Date
Leah Rowe ef867eccca git.sh: remove previous tmprepo if args>5
if doing a retry, the directory may still exist, which
would make git clone yield an error response; the existing
directory will have been the one that failed to reset, so
let's delete it.

the one deleted is not the cache (repo/PROJECT/), thus
otherwise maintaining current behaviour.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17 17:28:27 +01:00
Leah Rowe 4d5385a14d git.sh: try direct clone if cached git fails
normally, a project is cached at repo/PROJECT/, and
cloned from there to the final destination.

errors lead to a calling of $err, but this will result
in a return if done from inside a subshell, of non-zero
value, so use this to re-try with a 6th argument when
calling tmpclone().

in most cases, this fallback will never kick in, but
it will kick in resetting or patching the cached clone
fails; specifically, we are interested in the reset part.

a given project name may change repositories in lbmk at
a given time. if this happens, and the old one is cached,
the overall result of this patch is that lbmk will fall
back to the old behaviour, where git urls are tried
directly, without caching.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17 17:09:34 +01:00
Leah Rowe 0cd52fc7fd git.sh: re-try git pull three times
mitigate jittery internet connections

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17 16:55:05 +01:00
Leah Rowe 0b9cd77fe7 trees: auto-delete+auto-rebuild if project changes
actual source code is not scanned, but config directories are
scanned. simply get the checksum of each file under config/
pertaining to a given project/tree, and also for the given
target. coreboot utilities are also handled.

if it changes, in any way, delete and re-build automatically.

such deletions should probably still be done manually, as part
of understanding the build system, but this change should make
the build system much easier to use during development.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17 16:34:34 +01:00
Leah Rowe 583502027e git.sh: don't download to src/project/project/
re-use repo/project/

this means that single- and multi-tree projects now
have a unified cached git repo location, as per the
new rules, thus saving on disk space usage.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17 13:20:51 +01:00
Leah Rowe 9f09728aca git.sh: cache git downloads to repo/
do it based on the URL, e.g. https://review.coreboot.org/coreboot
becomes repo/coreboot

the downside is if you have two projects with repo urls specifying
the same string at the end, but this isn't the case at the moment
and likely won't be the case, but it's a theoretical issue.

this saves on bandwidth when downloading identical submodule repos
between multiple trees within the same multi-tree project

for example, coreboot 3rdparty/vboot is no longer downloaded more
than once, instead cloned locally on subsequent downloads.

if repo/DIR exists, git-pull is attempted, but errors do not result
in a non-zero exit, by design.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17 13:01:12 +01:00
Leah Rowe 3263eeb6b6 trees: general code cleanup
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-10 23:11:23 +01:00
Leah Rowe c0a4df688c trees: use wrapper for dry-running make commands
instead of using lots of if/else conditions, do that once
and set a variable, dry, to :

if not doing a dry run, the variable is empty. prefix this
variable in places where you don't want a certain action to
be performed, on dry runs.

more specifically, : does *nothing* and always returns with
zero status (success).

this results in cleaner code, and a small sloccount reduction.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-10 21:00:16 +01:00
Leah Rowe e9f66ec4a8 remove executable permission on include/
files under include/ should never be executed directly

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-10 02:14:25 +01:00
Leah Rowe 91994b7852 rom.sh: actually say if a cbutil exists
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-10 02:04:25 +01:00
Leah Rowe 97b777715b rom.sh: avoid re-building cbutils if built
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-10 01:37:27 +01:00
Leah Rowe 494b94799e rom.sh: only before cbutils before coreboot
otherwise, due to the idiosyncratic nature of the coreboot
build system, the coreboot.rom gets wiped out.

cbutils is still handled by premake. ensure that payloads are
only inserted just after running the coreboot make command.

fixes a build issues introduced on 9020sff, previously unhandled.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-10 01:24:05 +01:00
Leah Rowe 5a1d2401c2 minor cleanup
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-09 03:09:49 +01:00
Leah Rowe 9646172145 trees: support -d (dry run) for custom build logic
-d does the same as -b, except for actually building
anything! in effect, it does the same as -f (fetch)
except that the resulting variable assignments will
not be recursive (as with -f).

if -d is passed, configuration is still loaded, defconfig
files are still cycled through, and more importantly:

helper functions are still processed.

the grub, serprog and coreboot helper functions have
been modified to return early (zero status) if -d is
passed.

this behaviour will be used to integrate vendor.sh
logic in with the trees script, for cases where the
user wants to only handle vendor files. e.g.:

./update trees -b coreboot x230_12mb

this would download the files as usual, build coreboot,
with those files, and then build the payloads. but:

./update trees -d coreboot x230_12mb

this would download the files, NOT build coreboot, and
NOT build the payloads.

this change increases the sloccount a bit, but i'm relying
on the fact that the vendor.sh script already re-implements
config handling wastefully; the plan is to only use trees.

for now, simply stub the same ./vendor download command.

there is one additional benefit to doing it this way:

this method is *per-kconfig* rather than per-target.
this way, one kconfig might specify a given vendor file
that is not specified in the other. although the stub
still simply handles this per target, it's done in premake,
which means that the given .config file has been copied.

this means that when i properly re-integrate the logic
into script/trees, i'll be able to go for it per-kconfig.

the utils command has been removed, e.g.
./update trees -b coreboot utils default

the equivalent is now:
./update trees -d coreboot default

this would technically download vendor files, but here
we are specifying a target for which no kconfigs exist;
a check is also in place, to avoid running the vendor file
download logic if tree==target

the overall effect of this change is that the trees script
no longer contains any project-specific logic, except for
the crossgcc build logic.

it does include some config/data mkhelper files at the top,
for serprog and coreboot, so that those variables defined in
those files can be global, but another solution to mitigate
that will also be implemented in a future commit.

the purpose of this and other revisions (in the final push
to complete lbmk audit 6 / cbmk audit 2) is to generalise as
much logic as possible, removing various ugly hacks.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-09 02:55:20 +01:00
Leah Rowe e850c06bd2 rom.sh: only make rom tarball if release=y
Signed-off-by: Leah Rowe <info@minifree.org>
2024-07-08 02:22:22 +01:00
Leah Rowe 2f3cc5d379 rom.sh: new file, to replace script/roms
stub it from the trees script. the way it works now,
there is less code in the build system.

./build roms

this is no longer a thing

./build roms serprog

this is also no longer a thing. instead, do:

./update trees -b coreboot targetnamehere

./update trees -b pico-serprog

./update trees -b stm32-vserprog

the old commands still works, which causes the new
commands to run

coreboot roms now appear in elf/, not bin/, as before,
but those images now contain payloads.

NOTE: to contradict the above: ./build roms is no
longer a thing, in that it's now deprecated, but
backward compatibility is present for now. it will
be removed in a future release.

./build roms list also still works! it will do:
./update trees -b coreboot list

also:
./update trees -b grub list
this is now possible too

if a target "list" is provided, for multi-tree sources,
the targets are shown.

there is another difference: seagrub roms are now seagrub_,
instead of seabios_withgrub.

seabios-only roms are no longer provided, where grub is also
enabled; only seagrub is used. the user can easily remove
the bootorder file, if they want seabios to not try grub first.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-07 21:25:07 +01:00
Leah Rowe b7f12ade03 vendor.sh: remove mkdirs()
merge it into the only calling function

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30 19:22:58 +01:00
Leah Rowe 0018600d6e git.sh: simpler for loop in git_am_patches()
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30 18:23:15 +01:00
Leah Rowe 5882056a27 git.sh: merge for loops in git_am_patches()
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30 18:16:34 +01:00
Leah Rowe 67421a21fd trees and git.sh: tidy up global variables
some of the variables only initialised in git.sh are
also used in the trees script, which is technically ok
because git.sh is included from the trees script, but
it makes more sense to declare them in the latter.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30 18:10:22 +01:00
Leah Rowe 55dbd72aaf git.sh: simplified initialisation of "loc"
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30 17:56:32 +01:00
Leah Rowe 5ba0433b5f git.sh: condense fetch_targets() a bit
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30 17:04:24 +01:00
Leah Rowe 4772186b19 git.sh: short git_prep command in fetch_targets()
loc is already set, and will correspond to the same
path, so we can quite conveniently use it.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30 17:02:44 +01:00
Leah Rowe 017fd8259e git.sh: merge prepare_new_tree with fetch_targets
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30 15:49:35 +01:00
Leah Rowe 7394960619 git.sh: remove duplicate "xtree" variable
it's already defined inside the trees script

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30 14:00:19 +01:00
Leah Rowe 789631ccb8 trees, multi: download bare project *before* trees
when downloading multi-tree projects, the rev can be reset
to HEAD instead of the actual rev for a given target. this
occurs when the bare repo (e.g. src/coreboot/coreboot) does
not exist and has to be downloaded first.

bare repository downloading does not rely on target.cfg, in
this context, only pkg.cfg, but it uses the same variable
names (e.g. "rev").

instead of using a separate variable name, thus increasing
code complexity (which is the exact opposite of what i want
to do), do the bare repository download first.

this means that the git.sh script is much cleaner now, for
multi-tree projects, in that it *only* copies the bare repo
then runs git_prep; in that context, the bare repo is cloned
directly by calling the relevant function from script/trees,
which is the same behaviour as when cloning single-tree
project sources.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30 00:40:30 +01:00
Leah Rowe a5e724e51e trees: unified multi-tree configuration handling
the same function that loads configurations for single-tree
projects has been merged with the function for multi-tree
configs in git.sh, and that functionality has been removed
from git.sh; now it is all unified in the trees script.

as the saying goes: write one program to do one thing well.

the purpose of git.sh is to download source code, but not
to handle configuration files; the latter is meant to be
handled by the trees script, which then calls into git.sh
before running the build logic for that given project.

additionally: the "seen" files are no longer handled, at all.
the logic there was added ages ago, because at the time, i was
considering whether to separate configuration into a new
repository, so that users could more easily make their own
configuration, so it was a guard against misconfiguration.
however, that decision was canceled and we're always very
careful not to introduce a loop; if a loop does occur, the
worst that can possibly happen is you waste some CPU cycles.

Instead, print (on standard output) what config file is being
used, so the operator can see when an infinite loop occurs.

ALSO:

remove _setcfgarg in load_project_config()

it was used to skip when a target.cfg file didn't exist,
specifically on single-tree projects, but this is now
handled using -f instead, on the while loop inside that
function, so _setcfgarg is now a redundant variable.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30 00:02:03 +01:00
Leah Rowe 7bf2835207 git.sh: rename Fetch_project_trees fetch_targets
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 19:59:16 +01:00
Leah Rowe e0c244f37b git.sh: rename fetch_project_repo to fetch_project
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 19:58:03 +01:00
Leah Rowe d1b7882c5a vendor.sh readcfg: split the -b coreboot command
just add a line break to make the code more readable

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 18:55:41 +01:00
Leah Rowe 088a79918b vendor.sh: remove unnecessary check
the next command is a copy, which would give us the error
if the file doesn't exist, and an appropriate message

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 03:47:57 +01:00
Leah Rowe 8c5f78d3d3 vendor.sh: condense inject() a bit
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 03:27:31 +01:00
Leah Rowe 42925ce79f vendor.sh extract_kbc1126: use quote on file check
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 03:09:03 +01:00
Leah Rowe ddcc74777e vendor.sh extract_kbc1126ec: simplify build check
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 03:06:58 +01:00
Leah Rowe 7a12cd7704 vendor.sh: simplify e6400 vga rom file check
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 02:25:22 +01:00
Leah Rowe 163bcfcdfd vendor.sh: simplify variable checks for e6400vga
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 02:23:02 +01:00
Leah Rowe cbac2087cc vendor.sh: condense fetch() a bit more
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 02:21:00 +01:00
Leah Rowe 675d2c0376 vendor.sh: remove unnecessary check
this is over-engineering, because we do not allow just
about any path to be provided; it's not provided as an
argument in a command, for example.

this is dictated by a configuration file, which we control.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 02:14:00 +01:00
Leah Rowe 5e46b9f434 vendor.sh: simplify kbc1126 util handling
we don't need to check whether the binary exists, because
make already does that for us.

we still need to check that the directory exists, because
older versions of coreboot did not include kbc1126, and we
do still use older coreboot revisions on some boards.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 02:12:40 +01:00
Leah Rowe 7086b68685 vendor.sh: simplify bootstrap() utils handling
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 02:07:16 +01:00
Leah Rowe 6a05487ef3 vendor.sh: simplified initialisation of _7ztest
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 01:57:03 +01:00
Leah Rowe 972e61070b vendor.sh: condense detect_board() a bit
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 01:47:14 +01:00
Leah Rowe bf5acedb12 vendor.sh: condense patch_rom() a bit
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 01:41:48 +01:00
Leah Rowe 8b8cf070d3 vendor.sh: add return to end of bootstrap()
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 01:27:42 +01:00
Leah Rowe d85b9de2d9 vendor.sh: rename getcfg() to readkconfig()
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 01:26:31 +01:00
Leah Rowe 53b69f673c vendor.sh: merge cfgutils() into readcfg()
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 01:24:13 +01:00
Leah Rowe 55677d8217 vendor.sh: remove build_dependencies_inject()
merge it into vendor_inject()

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 01:22:28 +01:00
Leah Rowe c764198619 vendor.sh: simplify initialisation of variables
also unify handling of cbutils

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 01:19:29 +01:00
Leah Rowe d63da943c1 vendor.sh inject: skip serprog targets
this is to prevent a fault condition during automated testing

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 00:33:31 +01:00
Leah Rowe c6df9666af vendor.sh: don't inject /dev/null
if injection is attempted, verification comes next,
and verification fails.

this happens for kcma/kgpe amd boards, where pike2008
fake roms are inserted by inserting the correct pci
ids using /dev/null as a source. an empty pike2008 rom
prevents seabios from loading the real pci rom, and this
is done because the real one hangs SeaBIOS.

a similar fix was made for ./vendor download, but
overlooked for ./vendor inject. this patch fixes that.

Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29 00:23:26 +01:00