The new v3 applets don't use it, and eventually all applets will
use the new formats only.
This prepares the code for this, and moves trust, id_cache, and
root_fd to apk_ctx which is enough for v3 applets at this time.
The generic code is made to not initialize apk_database if it's
not needed.
This is a flat buffers inspired format that allows fast
mmaped access to the data with low overhead, signature support
and relatively good forward support.
On some systems the `/var/` dir is mounted in a tmpfs which is reseted
after each reboot. For that reason no post-install script can handle the
creation of the cache dir at `/var/cache/apk`.
Check on database opnening if the folder is available, if not create it.
Fixes#10715
Signed-off-by: Paul Spooren <mail@aparcar.org>
It used to be relative to the --root specified root, but that
causes issues with relative command line filenames and is unintuitive.
Update documentation accordingly. Fixes#10702.
This uses some macro trickery to make sure that there's one-to-one
mapping with the option index enum and the descriptor. The down
side is that enum's are generated via #define's and editors might
not pick them up for auto completion, but the benefits are more:
it's no longer possible have mismatching enum value and descriptor
index, and the amount of source code lines is less.
This greatly helps with memory management on applications that
may want to daemonize and open/close database several times.
Also the lifetime and "owner" of memory for all data is now
explicitly bound to owning struct apk_database, which might
be helpful when writing language bindings. As side effect, the
interned "atoms" are unique only within what apk_database, so
comparing packages from different apk_database may not work
as expected.
Fixes#10697
When creating an index apk warns if a dependency is missing a provider.
However when using a multi-arch repository, it's not an error that
a certain architecture is missing a dependency because that dependency
could be in an other architecture. Since apk index doesn't know about
this, add an argument to disable that warning.
Maintainer note: rebased for new option handling, and minor stylistic
adjustments.
Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
This reduces the number of relocations on PIE binaries, and also
reduces the executable size. Parsing of the options is slightly
sped up as only the exact matching option group parser is called.
This creates main help like:
--
usage: apk [<OPTIONS>...] COMMAND [<ARGUMENTS>...]
Package installation and removal:
add Add packages to WORLD and commit changes
del Remove packages from WORLD and commit changes
System maintenance:
fix Check WORLD against the system and ensure consistency
update Update repository indexes
upgrade Install upgrades available from repositories
cache Commands related to the management of an offline package cache
Querying package information:
info Give detailed information about packages or repositories
list List packages matching a pattern or other criteria
dot Generate graphviz graphs
policy Show repository policy for packages
Repository maintenance:
index Create repository index file from packages
fetch Download packages from global repositories to a local directory
manifest Show checksums of package contents
verify Verify package integrity and signature
Miscellaneous:
audit Audit directories for changes
stats Show statistics about repositories and installations
version Compare package versions or perform tests on version strings
This apk has coffee making abilities.
--
And applet specific help like:
--
usage: apk add [<OPTIONS>...] PACKAGES...
Description:
apk add adds the requested packages to WORLD and installs (or upgrades)
them if not already present, ensuring all dependencies are met.
Options:
--initdb Initialize a new package database
-l, --latest Disables normal heuristics for choosing which repository to install a
-u, --upgrade When adding packages which are already installed, upgrade them rather
-t, --virtual NAME
Instead of adding the specified packages to WORLD, create a new
--no-chown Do not change file owner or group
--
If apk is run as a non-root user, it's not possible to chown files.
Maintainers note: minor wording changes on commit log and man page.
Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
We want to include via <apk/$headerfile> and not via <$headerfile>,
so we want to add an include statement for the dir the apk folder
which contains the headers is contained in.
Most notably this happens after blank database is initialized with --initdb,
but can happen also in other conditions.
The error checking changes modified behaviour if the file does not exist.
Explicitly check for ENOENT and ignore it. But the behaviour is improved
from earlier as now e.g. EPERM will be detected and reported correctly.
Fixes#10679
Fixes: 6da3e8eb15 "istream, archive, db: convert db and tar function to use istream"
Some screen size are quite small. For example, the default phosh
terminal is less than 50 character wide on Pinephone. This lowers the minimum
loading bar size to 25 characters.
For comparison, 25 character wide is just as wide as "apk add firefox
linux-lts" without the quotes.
Here's a bad picture to illustrate the result
gitlab.alpine.org/uploads/48c20f746fbf685b62b6bd73585ecbf2/pinephone-phosh.png
- always create .so and .a with the apk libary code
- create .pc file
- install headers and above mentioned files
Maintainers edit: merged commits, cleaned up sed script logic,
and few other minor fixes.
This enforces all scripts to be in the control block, and
all data files to be in data block. Ignoring of dot files in
root is added back: packages without any real files will
ship one ".dummy" item in the data block to trigger processing
and validation to work.
This mostly boils down to making sure control_started and
data_started are consistently used to gate actions, instead of
relying whether on file names start with a '.'.
None of the weaknesses this fixes are exploitable, but they
might have become so after changes to seemingly-unrelated code,
so it's good to clean them up.
This change ensures that apk_sign_ctx_process_file() and
apk_sign_ctx_parse_pkginfo_line() are called during archive
processing, allowing discovery of signatures and the data section
checksum.
Fixes a bug uncovered by commit f123d77e.
Convert all implementations to do buffering. This is in preparation
to remove bstream interface as redundant.
istream_read() will return full reads unless end-of-file. The backends
can return short reads to optimize buffering or due to other reasons
like boundary change for gz.
The "apk info" option --install-if, which had the short option -i,
was being shadowed by the apk global option --interactive, which
has the same short option. To fix this issue the short option for
--install-if was removed, as has been done in other cases where an
applet short option would be shadowed by a global short option. The
short option for --rinstall-if was also removed for consistency,
since it is the complement of --install-if.
This commits adds a function for receiving the closet byte unit and the
size in that unit for a given size_t. The function doesn't return a
string since dynamic memory allocation (or a static buffer) would be
required to implement this properly.
Discussion: It might be useful to add a command line flag for disabling
this behaviour in order to retain compatibility with the previous apk
info output format.
This reverts commit d379edd5bf.
Requested multiple times. Let's enable this for now. For v3.0,
we reconsider which applets to keep and remove the ones not needed.
When unpacking a file that is in root, it got a temporary file
name /.apk... however if the --root option was used it should
have the name root/.apk... otherwise unpacking will fail if the
user does not have write access to /.
Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Since 37fbafcd92 the tests hasn't worked
since a string on the stack was added as version for a virtual package.
Instead create the version string on the heap and then simply leak it.
A short running program as apk shouldn't need to worry about memory
leaks.
Fixes 37fbafcd by adding more input to the hash than just second
grained time stamp - collisions would happen when running apk
scripted.
For virtual package the hash works only as unique identifier, so
try to add elements that should make it unique in most cases.
Fixes#10648
Add also some testing to make sure help, long help and handling
of invalid arguments works as expected.
Based on pull request #19 originally by Laurent Arnoud (@spk).
The return -1 seems to have been left over from earlier code, and
could have been treated as -EPERM. This helps to fix the other command
line handling that potentially require changing.
Notably this fixes occasional issues when doing upgrade with multiple
versions of same packages. Without this the upgrade flag is not always
propagated properly down the dependency chain.
Originally the virtual packages could have dependencies added to it.
However, commit b06e3b99 broke this behaviour to fix error reporting.
The root cause however was that the virtual depedency package was not
properly versioned.
This fixes to use current date/time as the package version, and
constructs the "faked" package hash from it. This effectively makes
"add -t virtpkg deps.." replace the dependencies which should be the
desired behaviour for "abuild deps".
'world' dependency to the generated virtual package is also now
versioned to make sure it get's upgraded.
BIT(1) corresponds with decimal 2, which is the first available repository.
Before this fix, `apk list -O` would list every package installed from the
first available repository, which is the 'system' repository on most Adélie
Linux computers.
After this fix, `apk list -O` correctly lists only the packages which are
no longer available.
merge_options() will write one more entry to the options table
which is the end-of-table indicator. Allocate memory for it too.
valgrind did not pick it up due to being in stack; changing alloca
to malloc would make valgrind notice the issue too.
Reported-by: Mobile Stream <info@mobile-stream.com>
A crafted .apk file could to trick apk writing unverified data to
an unexpected file during temporary file creation due to bugs in handling
long link target name and the way a regular file is extracted.
Several hardening steps are implemented to avoid this:
- the temporary file is now always first unlinked (apk thus reserved
all filenames .apk.* to be it's working files)
- the temporary file is after that created with O_EXCL to avoid races
- the temporary file is no longer directly the archive entry name
and thus directly controlled by potentially untrusted data
- long file names and link target names are now rejected
- hard link targets are now more rigorously checked
- various additional checks added for the extraction process to
error out early in case of malformed (or old legacy) file
Reported-by: Max Justicz <max@justi.cz>
Most applets return whatever apk_solver_commit() returns. It is the
number of errors found (or negative for hard error). Sanitize the
error value to not give false success exit code in the unlikely case
of errors % 256 == 0.
Reported-by: Max Justicz <max@justi.cz>
It's common for a pre-install script to do something like
addgroup -S group 2>/dev/null
When apk installs files after this, it sets the owner/group based on id cache
but currently the id cache is stale and doesn't contain the new group at that
point: instead the file will be installed with gid that the build host
happened to have for that group -- on target this might mean a non-existing
group or a completely different group.
We can't know if the script really did modify id cache contents so make sure
to reset the id cache on every script execution.
Virtual packages have the origin pointer set to NULL. Trying to print it
using the BLOB_PRINTF macros causes a segmentation fault.
Inspired by the `print_origin_name` function from `src/search.c` this
commit attempts to fix it by checking whether `pkg->origin` is NULL
before attempting to print it. If it is NULL the pkg name is printed
instead.
Since printing the pkg name requires a different format string this
commit splits the printf call for printing the package line into
multiple ones. The output format shouldn't have changed at all though.
Run apk_solver_solve once with all args as dependencies instead of
running apk_solver_solve for each arg. This is neccesary so the
install_if calculation is done correctly.
During netboot on systems without RTC, time() will be near zero,
and the index fill not exist. Thus the plain test of st.st_mtime
against system time failed. Verify that fstatat() succeeds.
The list applet provides a convenient way of inspecting both the available
and installed package databases by listing their contents. In some ways,
it is similar to `apk search` but is considered to be a superset of
`apk search` functionality.
A few `apk list` criterion are not yet ready though, such as `apk list --depends`
which searches by runtime dependency (replacing `apk info --rdepends`).
This modifies apk cache for indexes to be automatically refreshed
periodically without explicit 'update' or '--update-cache' usage.
The default is to do if-modified-since request if the local copy
is older than 4 hours. This age can be changed with --cache-max-age.
Using --update-cache will change this age to 60 seconds to make
sure the cached copy is relatively new. The small age is in order
to try to avoid downloading indexes second time when apk-tools is
upgraded and apk re-execs after self-upgrade.
Accordingly using explicitly 'apk update' will now enforce
--force-refresh and request the very latest index by requesting
any potential http proxy to do refresh too.
This unloads --force as several of the things are really not wanted
together. E.g. --force-refresh is a lot different from --force-broken-world
and doing --force to get the other might introduce unwanted behaviour.
--force is still kept for backwards compatibility and it enables
most things --force was used for.
This flag skips running hook scripts
This flag *must* be used during initramfs tmpfs initial install.
The reason that this new flag is needed is that the hooks will currently
always fail as musl and /bin/sh is missing at this stage on diskless.
This change just changes to keep deleted directory items in
the hash with ref count zero and modified flag set. Those entries
are reused when needed. The side effect is that fire_triggers()
will now see those removed direcotries and reports them. Other
enumerators of the directories hash are protected to skip removed
directories when appropriate.
In discovery phase, there was logic to not process packages
multiple times. However, that logic failed to account the package's
depth and install_if state for the name being processed. This
caused install_if processing failure in certain topologies of the
dependency graph. Adds also a test case that should catch this
issue reliably.
By introducing a new package metadata field, `provider_priority`
(index letter `k`), we can specify default packages to satisfy a
virtual.
If a user wishes to select an alternative provider for the virtual,
a changeset swapping the default provider for the selected provider
will be generated by the dependency resolver.
this fixes package selection when a 'real' package exists, but would
need to be provided by another package with 'virtual provides'.
In current package database this can happen with postgresql which is
also provided by postgresql-bdr. Normally postgresql would be satisfied
by postgresql, but if any package depends on postgresql-bdr and there's
no versioned dependency on postgresql this will help apk figure out
that postgresql-bdr should be used.
APKs have been created with GNU tar so far, which uses the
GNU extensions for long names. In order to increase portability
support the standard header's 'prefix' portion in case
the GNU extensions are not present.
Depending how the directory entries are ordered, the cached dir
instance might not have been updated correctly. This has not been
a problem as the entries have been ordered, but is now triggered
on ppc.
The value from tar header is unsigned int; keep it casted to
unsigned int and size_t instead of (signed) int, otherwise
the comparisons fail to do their job properly. Additionally check
entry.size against SSIZE_MAX so the rounding up later on is
guaranteed to not overflow.
Fixes CVE-2017-9669 and CVE-2017-9671.
Reported-by: Ariel Zelivansky from Twistlock
- rework the progress bar a little bit, basically removing the [ and ] to give a more
modern aesthetic.
- if utf-8 locale is enabled, use unicode codepoint 0x2588 instead of # to give the
progress bar a nicer look.
- if APK_PROGRESS_CHAR environment variable is defined, use the character defined there
for the progress bar.
This is useful for requirements such as: python3=~3.6, which would match python3-3.6.[0-9].
This implementation should in theory be backwards compatible with the implementation in Adelie.
Fixes regression from commit 5ba27c90 which caused stdio
buffering issues now that output is split to stderr/stdout.
See also commit 51737872 for some of the history.
Fixes#7107
On some (probably buggy) terminals, printing up to the rightmost column
may end up with the terminal issueing a newline (probably due to putting
the cursor at the next char -> new line).
Some other progress bars avoid it by not reaching the rightmost column.
Shorten the bar width from term-width - 7 to -8 (the 7 are "xxx% []").
fixes#5616
The original intention was not use unnecessary space on tmpfs
e.g. if the cache directory is a mount point, but accidentally
left unmounted. But there are valid cases when packages are
intentionally wanted to be cached on tmpfs. If caching is not
desired, the user can just remove the cache directory.
This allows for instance integration of etckeeper
[TT: Reorganized code a bit, and modified to use single
directory commit_hooks.d with argument for script of stage.]
Currently apk only knows about powerpc and ppc64. I am adding support for ppc64le.
ppc64le is the based on the ABI v2, which defines the endianess as little,
while ppc64 is based on the first 64-bits ABI.
It is unreasonable to assume that all package writers would except
to reset umask themselves. It's done currently in most packages,
but we had first issue of this kind recently, so better just reset
umask.
Make it possible to individually override openssl, zlib and libfetch
cflags and linker flags. This makes it possible to build apk-tools
without having pkg-config installed.
it's only used to speed up things, and having it non-existant
is not a fatal error - all included things in index will be
passed in command line anyway
which will delete any .apk package on output directory that were
not downloaded by fetch
this allows apk fetch to incrementally build repositories for
binary images
- self-conflicts when the exact same version of a name is provided
twice is now properly detected and diagnozed
- don't print redundant satisfies diagnostic
Implement --no-cache. The index is read directly from network and not
cached. This is useful for docker, where you install a set of packages
and directly after purge the cache. (see
1fc9e59d16/builder/scripts/apk-install)
fixes#4905
preserve [am]time for all packages and indexes. this fixes the caching
error that 'apk update' is after new index is generated, but before
the used mirror is synchronized. this caused local apkindex timestamp
to be newer than file in mirror, when in fact it was outdated index.
this also fixes fetched files to have build timestamp so that files
going to .iso or custom images have proper timestamps (rsync with
appropriate --modify-window now works)
the problem is that var/lock is on root installs symlink to /run/lock
(on tmpfs) and does not exist if doing chroot() to that root. fixes
apk to work when chrooted to existing rootfs install.
resolve reverse dependencies after all packages have been loaded,
and avoid traversing the reverse name lists. now that we use
automatic virtual packages (soname, pkg-config, etc.) the reverse
dependency chains can become considerable longer than what it was
when the rdependency construction code was originally written.
In practice this should fix to e.g. not wipe out /etc/apk/world if
final flush to /etc/apk/world.new fails.
This was prompted by an incident the other day where I ran the root
partition of an Alpine box out of space using 'apk add', and apk
helpfully wiped the contents of /etc/apk/world at the same time.
It might be tricky to try to reproduce exactly the same failure,
but from an examination of the code, setting 'rc' before the final
call to fdo_flush rather than after is one possible cause of this
behavior. (If the entire contents of /etc/apk/world.new are buffered,
and all get written out in the final fdo_flush call, and that call
fails, fdo_close will still happily rename /etc/apk/world.new to
/etc/apk/world.)
Timo Teräs
Ariadne Conill