Commit Graph

91 Commits (b43da45bc31a5d09fc03e8bb1ce7c3f875bdfaaa)

Author SHA1 Message Date
thibault.ferrante 18b0b45b5b io: Handle long lines
As an APKINDEX can become arbitrarely long due to dependencies and
other parameters, increasing the buffer size make this less likely
to happens.

Closes #10731.
2021-01-11 11:26:49 +02:00
thibault.ferrante 0fb0d30477 database: Propagate errors when loading an APKINDEX
In case of failure when loading an APKINDEX, no errors are
propagated to the user which may uncorrectly interpret the
current problem.
2021-01-11 11:26:49 +02:00
Timo Teräs a627ab8eb8 make apk_database optional for applets that don't need it
The new v3 applets don't use it, and eventually all applets will
use the new formats only.

This prepares the code for this, and moves trust, id_cache, and
root_fd to apk_ctx which is enough for v3 applets at this time.

The generic code is made to not initialize apk_database if it's
not needed.
2020-10-09 16:09:19 +03:00
Timo Teräs 6d71f49c11 io: make ostream_file always use tmpname 2020-10-09 16:09:19 +03:00
Timo Teräs efe0c4afec adb: introduce apk-tools database format, and few applets
This is a flat buffers inspired format that allows fast
mmaped access to the data with low overhead, signature support
and relatively good forward support.
2020-10-09 16:09:19 +03:00
Timo Teräs d0edeec8fb make the atom functions not use global state
This greatly helps with memory management on applications that
may want to daemonize and open/close database several times.

Also the lifetime and "owner" of memory for all data is now
explicitly bound to owning struct apk_database, which might
be helpful when writing language bindings. As side effect, the
interned "atoms" are unique only within what apk_database, so
comparing packages from different apk_database may not work
as expected.

Fixes #10697
2020-05-19 12:02:56 +03:00
TBK 5d796b5678 use SPDX-License-Identifier in source files 2020-05-07 10:45:42 +03:00
Timo Teräs 72be813930 io: add stream copy helper 2020-02-14 16:45:13 +02:00
Reid Rankin 6cc3e6a1de don't ignore md parameter to apk_fileinfo_hash_xattr_array() 2020-01-25 16:33:17 +02:00
Timo Teräs 67696b2ac6 io: use min() instead of MIN() 2020-01-11 11:32:51 +02:00
Timo Teräs 1de9ef422c io: convert bstream mmap to istream, remove the now obsolete bstream machinery 2020-01-11 11:32:21 +02:00
Timo Teräs 7caa217731 convert remaining locations to use istream instead of bstream 2020-01-11 11:20:48 +02:00
Timo Teräs 6da3e8eb15 istream, archive, db: convert db and tar function to use istream 2020-01-11 11:20:45 +02:00
Timo Teräs 7ca0d146ec istream: add buffering capability
Convert all implementations to do buffering. This is in preparation
to remove bstream interface as redundant.

istream_read() will return full reads unless end-of-file. The backends
can return short reads to optimize buffering or due to other reasons
like boundary change for gz.
2020-01-11 03:44:23 +02:00
Timo Teräs 3cd7d1e077 io: remove the now unused pid association with istream 2020-01-06 01:17:08 +02:00
Timo Teräs e39334e44f io: remove unused size parameter from bstream close 2020-01-06 00:27:17 +02:00
Timo Teräs 6996b1ea75 io: use proper base struct types for method implementations 2019-12-18 10:00:29 +02:00
Timo Teräs f38d1f74af fix xattr hash to be sha1
The hash type was accidentally changed in previous commit. Currently
csum->data cannot hold longer hash, so fix the hash.
2018-10-30 18:26:10 +02:00
Timo Teräs beab8545eb add support for openssl 1.1 2018-10-26 08:22:08 +03:00
Timo Teräs 8a28c6d0d4 enable automatic update of indexes controlled by --cache-max-age
This modifies apk cache for indexes to be automatically refreshed
periodically without explicit 'update' or '--update-cache' usage.

The default is to do if-modified-since request if the local copy
is older than 4 hours. This age can be changed with --cache-max-age.
Using --update-cache will change this age to 60 seconds to make
sure the cached copy is relatively new. The small age is in order
to try to avoid downloading indexes second time when apk-tools is
upgraded and apk re-execs after self-upgrade.

Accordingly using explicitly 'apk update' will now enforce
--force-refresh and request the very latest index by requesting
any potential http proxy to do refresh too.
2018-01-04 10:56:09 +02:00
Timo Teräs 2f3c842049 io: fix skip and splice to detect unexpected end-of-file 2017-10-12 13:35:46 +03:00
Dmitry Golovin 16336ba265 fix comparison of unsigned expression < 0 is always false
found by clang
2017-08-23 08:56:44 +03:00
Timo Teräs 4d9c0c39b0 io: make io vtables const struct, and add accessors for them
This reduces function pointers in heap, and unifies how the
io functions are called.
2017-06-23 10:07:44 +03:00
Henrik Riomar 349c61c961 add support for pre and post commit hooks
This allows for instance integration of etckeeper

[TT: Reorganized code a bit, and modified to use single
directory commit_hooks.d with argument for script of stage.]
2017-02-15 13:44:04 +02:00
Timo Teräs cce4cff553 io, database: preserve [am]time for cached and fetched files
preserve [am]time for all packages and indexes. this fixes the caching
error that 'apk update' is after new index is generated, but before
the used mirror is synchronized. this caused local apkindex timestamp
to be newer than file in mirror, when in fact it was outdated index.

this also fixes fetched files to have build timestamp so that files
going to .iso or custom images have proper timestamps (rsync with
appropriate --modify-window now works)
2015-11-09 12:51:01 +02:00
Natanael Copa 707b51e0c6 io: fix posix_fallocate failure handling
We need fall back to a splice buffer if posix_fallocate call fails due
to file being a device (eg tty) or a pipe. This fixes apk fetch --stdout.
2015-10-08 11:42:23 +03:00
Timo Teräs 9ffa38222b io: use posix_fallocate to allocate disk space
ftruncate does not allocate it, and subsequent access to mmaped
file will result in SIGBUS. this fixes to properly report disk
full errors.
2015-09-03 13:15:18 +03:00
Timo Teräs 6b140ec5c7 fix bstream_from_fd to use mmap when available 2015-06-12 09:57:52 +03:00
Alex Dowad 4c3712ecb4 detect failures in writing to file during final flush of buffers
In practice this should fix to e.g. not wipe out /etc/apk/world if
final flush to /etc/apk/world.new fails.

This was prompted by an incident the other day where I ran the root
partition of an Alpine box out of space using 'apk add', and apk
helpfully wiped the contents of /etc/apk/world at the same time.

It might be tricky to try to reproduce exactly the same failure,
but from an examination of the code, setting 'rc' before the final
call to fdo_flush rather than after is one possible cause of this
behavior. (If the entire contents of /etc/apk/world.new are buffered,
and all get written out in the final fdo_flush call, and that call
fails, fdo_close will still happily rename /etc/apk/world.new to
/etc/apk/world.)
2015-05-26 08:38:45 +03:00
Alex Dowad be31eb24d8 io: fix compiler error by including stdint.h
According to the C standards, uint32_t is defined in stdint.h.
Presumably apk is usually built against C libraries where
stdint.h is indirectly included through another header file,
but this isn't the case with the version of glibc which I am using.
2015-04-22 10:50:31 +03:00
Timo Teräs 3490ff789a fix tee io error handling
use ERR_PTR mechanism, and handle it at all places.
2015-04-13 09:42:27 +03:00
Timo Teräs 60dd5798c9 use memmove for copying buffer leftovers, as the ranges may overlap
issue cought by fortify
2015-04-08 16:58:20 +03:00
Timo Teräs 9ed5a60dec unbreak audit after xattr support 2015-04-08 11:16:46 +03:00
Timo Teräs 83ab022301 audit xattrs
ref #3027
2015-04-08 10:27:49 +03:00
Timo Teräs 8d1ec4c5bc calculate and store checksum of xattrs
ref #3027
2015-03-11 16:10:33 +02:00
Timo Teräs da6e455f70 rename file info related functions for consistency 2015-03-10 15:47:13 +02:00
Timo Teräs be8e133c0b extract xattrs from packages
ref #3027
2015-03-10 14:38:06 +02:00
Timo Teräs 255fd81d79 rework error handling for write streams 2015-03-10 13:15:58 +02:00
Timo Teräs 2a6896b2b4 rework error handling for read streams 2015-03-10 13:15:31 +02:00
Timo Teräs c553970061 remove support for old database location in /var
the location changed in apk-tools 2.1.0 (March 2011) which was
used in Alpine Linux 2.2.
2015-01-30 14:35:02 +02:00
Timo Teräs 32627939f5 io,url,db: support for if-modified-since 2014-10-08 11:13:21 +03:00
Timo Teräs 1bbca16333 io: fix few error path leaks 2014-10-07 17:04:38 +03:00
Timo Teräs 26558bd126 io: use fget{pw,gr}ent_r only on uclibc and glibc
musl does not have those.
2013-07-04 10:36:47 +03:00
Timo Teräs 656190b0a0 io: get rid of APK_PROGRESS_SCALE
no need to muldiv all the time, just pass the current amount of bytes
done, and let callbacks use that directly.
2013-06-17 17:28:03 +03:00
Timo Teräs ade8d0b4e9 cache: implement progress bar (ref #1170) 2013-06-17 17:13:14 +03:00
Timo Teräs dbb642206d io: fix splice for copying unknown lengths 2013-06-17 14:23:14 +03:00
Timo Teräs 7392acb95e db: keep architecture in $ROOT/etc/apk/arch
This we use proper arch in case modifying chroot installation.
2012-02-23 17:04:51 +02:00
Timo Teräs ba3ee3f863 audit, db: add etc/apk/protected_files.d and new audit features
fixes #607.

audit is now mostly rewritten for the new functionality. And
has new features like --check-permissions, --recursive and
--packages.

$ROOT/etc/apk/protected_files.d/*.list can now contain additional
protected paths, one path per line:

+etc
@etc/init.d
-tmp

+ will include the directory as protected configuration directory.
@ considers the directory protected, but will backup only symlinks.
- removes any protection

lbu should be modified to put include and exclude paths in
etc/apk/protected_files.d/lbu.list. Additionally, some packages
might provide their own listings.

E.g. ssh might want to provide ssh.list with something like:
+root/.ssh
+home/*/.ssh
2012-02-23 15:05:06 +02:00
Timo Teräs 77e203bf32 db, io: load repositories also from etc/apk/repositories.d/*.list
Load additional repositories from $ROOT/etc/apk/repositories.d/*.list
unless --repositories-file is given as parameter.
2012-02-22 13:57:05 +02:00
Timo Teräs bf82e2e5fd db, solver, io: scan cache items at startup
It is faster to just scan the cache directory for existing packages
at startup than trying to faccessat() them on demand. It also makes
quite a few parts of the code more readable and simpler.
2012-02-22 08:45:40 +02:00