If server redirects from http to https, libfetch detects this, but
wrongly uses the old url scheme to determine the port. This subsequently
leads to the following OpenSSL error:
139741541575496:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:331:
Using the new scheme fixes this. This error message comes from trying
to connect to port 80 with TLS, it can also be observed by issuing
$ openssl s_client -connect alpinelinux.org:80
This bug was introduced in commit:
7158474 libfetch: keep http auth only if redirect is for the same host
(De)initialization is deprecated under OpenSSL 1.0 and above.
[TT: Some simplifications, and additional edits.]
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Treat URLs with too long individual components as malformed instead
of silently truncating that field. There might be unexpected results
if hostname, username or password field gets truncated.
The connection pooling was broken in two ways:
1. The original URL was always used as the connection pool URL,
resulting in duplicate connections to the proxy for http URLs
(each http URL would get separate proxy connection)
2. The cache_url stored was always the socket level connect URL.
In case of HTTPS, the lookup was done done with the real URL,
but the proxy URL was stored as the "cache URL". Thus HTTPS
CONNECT connections were never re-used.
This fixes the code with following logic:
1. The cache key url is the real URL when no-proxy, or when HTTPS
with proxy (the socket is connected to proxy, but logically it
is connected to the real URL due to HTTP CONNECT request).
And for HTTP with proxy, it's the proxy URL so same proxy
connection can be reused for all requests going through it.
2. fetch_connect() now gets cache key URL separately, and it always
gets the same value as the fetch_cache_get() calls.
This enables matching numeric hosts against CIDR style subnet
matches in no_proxy environment variable.
Based on work by Thermi.
Co-authored-by: Noel Kuntze <noel.kuntze@thermi.consulting>
This creates main help like:
--
usage: apk [<OPTIONS>...] COMMAND [<ARGUMENTS>...]
Package installation and removal:
add Add packages to WORLD and commit changes
del Remove packages from WORLD and commit changes
System maintenance:
fix Check WORLD against the system and ensure consistency
update Update repository indexes
upgrade Install upgrades available from repositories
cache Commands related to the management of an offline package cache
Querying package information:
info Give detailed information about packages or repositories
list List packages matching a pattern or other criteria
dot Generate graphviz graphs
policy Show repository policy for packages
Repository maintenance:
index Create repository index file from packages
fetch Download packages from global repositories to a local directory
manifest Show checksums of package contents
verify Verify package integrity and signature
Miscellaneous:
audit Audit directories for changes
stats Show statistics about repositories and installations
version Compare package versions or perform tests on version strings
This apk has coffee making abilities.
--
And applet specific help like:
--
usage: apk add [<OPTIONS>...] PACKAGES...
Description:
apk add adds the requested packages to WORLD and installs (or upgrades)
them if not already present, ensuring all dependencies are met.
Options:
--initdb Initialize a new package database
-l, --latest Disables normal heuristics for choosing which repository to install a
-u, --upgrade When adding packages which are already installed, upgrade them rather
-t, --virtual NAME
Instead of adding the specified packages to WORLD, create a new
--no-chown Do not change file owner or group
--
Fix comparing of the hostname portion that matches exactly.
The no_proxy matching is pretty rudimentary though and probably
could go through a bit of additional rework.
Fixes#10681
- split the code to a helper function
- do not set sockets to corked state when putting back to
cache so socket state is always deterministic
- cork/uncork also when sending CONNECT to a proxy, this
can reduce a little bit the latency how fast the packet
gets sent out
- also pair corking with uncorking in http_request to make
it more obvious pairing
The recent TCP_CORK change missed this bit of code. This change
should improve performance a bit when making HTTP requests by calling
http_cmd only once instead of three times.
Unfortunately libfetch operates on raw sockets and is sending
each HTTP request line using separate syscall which causes the
HTTP request to be sent as multiple packets over the wire in most
configurations. This is not good for performance, but can also
cause subtle breakage if there's DPI firewall that does not get
the Host header.
Incidentally, it seems that on BSDs libfetch already sets
TCP_NOPUSH optimize the packetization. This commit adds same
logic for using TCP_CORK if available. When using TCP_CORK
there is no requirement to set TCP_NODELAY as uncorking will
also cause immediate send. Keep TCP_NODELAY in the fallback
codepaths.
Long term, it might make sense to replace or rewrite libfetch
to use application level buffering.
X509_check_host() is introduced in libressl-2.5.0 and openssl-1.0.2
which are not yet universally available. Add support for building
against the older versions.
loosely based on the freebsd implementation, implement https
connection settings to override CA, and use client certificate.
new files supported in /etc/apk/:
ca.pem - if exists, it contains CAs acceptable for https
(otherwise system wide CAs are used)
crl.pem - if ca.pem is used, this is the (optional) CRL for it
cert.pem - used as client authentication certificate (+ key)
cert.key - used as client key (can be also inside cert.pem)
ftp://ftp.fu-berlin.de/unix/NetBSD/packages/current-src/pkgsrc/net/libfetch/files
libfetch comes (at least) in netbsd and freebsd flavors which
differing functionality. Alpine and Arch package netbsd one,
but it's not widely packaged across other distributions.
We need NetBSD version as it does not use funopen(3) which is not
supported in musl, and supports connection pooling.
FreeBSD seems to be the orignal and better maintained version
with support for SSL CAs, client certificate authentication,
proxy authentication, and improved http redirect handling.
So this imports NetBSD version, and future commits will pick up
the needed improvements from FreeBSD tree.
Incidentally, this also fixes#7857 and likes for good.
Timo Teräs
Ariadne Conill