Commit Graph

34 Commits (9a9df378b70675f6b61d60d49ecab8be885fd855)

Author SHA1 Message Date
ThibG c297a1e7a7 Fix Ruby 2.7 support (#12831)
- update http gem to avoid errors
- update blurhash gem to avoid shared object loading error
- update goldfinger gem so the http gem could be updated
- update json gem to avoid warnings
2020-01-12 15:49:30 +01:00
Daigo 3 Dango 18cd3a9cb3 Suppress backtrace when delivering toots (#12798)
This is to suppress irrelevant backtrace from errors raised when
delivering toots to remote servers. The errors are usually out of
control by the local server and backtraces don't provide much
information.

This is similar to https://github.com/tootsuite/mastodon/pull/5174
and shortens backtraces like below:

```
WARN: Mastodon::UnexpectedResponseError: https://example.com/inbox returned code 523
WARN: app/workers/activitypub/delivery_worker.rb:48:in `block (3 levels) in perform_request'
app/lib/request.rb:75:in `perform'
app/workers/activitypub/delivery_worker.rb:47:in `block (2 levels) in perform_request'
app/lib/request_pool.rb:53:in `use'
app/lib/request_pool.rb:108:in `block (2 levels) in with'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/notifications.rb:170:in `instrument'
app/lib/request_pool.rb:107:in `block in with'
app/lib/connection_pool/shared_connection_pool.rb:21:in `block (2 levels) in with'
app/lib/connection_pool/shared_connection_pool.rb:20:in `handle_interrupt'
app/lib/connection_pool/shared_connection_pool.rb:20:in `block in with'
app/lib/connection_pool/shared_connection_pool.rb:16:in `handle_interrupt'
app/lib/connection_pool/shared_connection_pool.rb:16:in `with'
app/lib/request_pool.rb:106:in `with'
app/workers/activitypub/delivery_worker.rb:46:in `block in perform_request'
vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:51:in `run_code'
vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:42:in `run_yellow'
vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:24:in `run'
app/workers/activitypub/delivery_worker.rb:57:in `perform_request'
app/workers/activitypub/delivery_worker.rb:25:in `perform'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:31:in `block in call'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/statsd/publisher.rb:27:in `statsd_time'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:30:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
app/lib/sidekiq_error_handler.rb:5:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/scout_apm-2.3.0.pre3/lib/scout_apm/background_job_integrations/sidekiq.rb:69:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-unique-jobs-6.0.18/lib/sidekiq_unique_jobs/server/middleware.rb:29:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:73:in `block in wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:72:in `wrap'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
```

```
WARN: Stoplight::Error::RedLight: https://example.com/inbox
WARN: vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:46:in `run_red'
vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:25:in `run'
app/workers/activitypub/delivery_worker.rb:57:in `perform_request'
app/workers/activitypub/delivery_worker.rb:25:in `perform'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:31:in `block in call'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/statsd/publisher.rb:27:in `statsd_time'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:30:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
app/lib/sidekiq_error_handler.rb:5:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/scout_apm-2.3.0.pre3/lib/scout_apm/background_job_integrations/sidekiq.rb:69:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-unique-jobs-6.0.18/lib/sidekiq_unique_jobs/server/middleware.rb:29:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:73:in `block in wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:72:in `wrap'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
```
2020-01-11 02:15:03 +01:00
ThibG 5f6fd01ea6 Attempt to concurrently connect to remote IP addresses (#11757)
* Attempt to concurrently connect to remote IP addresses

* Reduce code length to please CodeClimate 🤷
2019-09-05 05:32:53 +02:00
Eugen Rochko 317b79d673 Add HTTP signatures to all outgoing ActivityPub GET requests (#11284) 2019-07-11 14:49:55 +02:00
Eugen Rochko 6baf5099a6 Refactor fetching of remote resources (#11251) 2019-07-10 18:59:28 +02:00
Eugen Rochko ad6b03d4b6 Fix URLs appearing twice in errors of ActivityPub::DeliveryWorker (#11231) 2019-07-07 03:37:01 +02:00
ThibG ac1fec5be3 Fix support for HTTP proxies (#11245)
* Disable incorrect check for hidden services in Socket

Hidden services can only be accessed with an HTTP proxy, in which
case the host seen by the Socket class will be the proxy, not the
target host.

Hidden services are already filtered in `Request#initialize`.

* Use our Socket class to connect to HTTP proxies

Avoid the timeout logic being bypassed

* Add support for IP addresses in Request::Socket

* Refactor a bit, no need to keep the DNS resolver around
2019-07-07 02:05:38 +02:00
ThibG 33607c43a5 Fix HTTP requests to IPv6 hosts (#11240) 2019-07-05 22:03:00 +02:00
Eugen Rochko e782fcef4f Add request pool to improve delivery performance (#10353)
* Add request pool to improve delivery performance

Fix #7909

* Ensure connection is closed when exception interrupts execution

* Remove Timeout#timeout from socket connection

* Fix infinite retrial loop on HTTP::ConnectionError

* Close sockets on failure, reduce idle time to 90 seconds

* Add MAX_REQUEST_POOL_SIZE option to limit concurrent connections to the same server

* Use a shared pool size, 512 by default, to stay below open file limit

* Add some tests

* Add more tests

* Reduce MAX_IDLE_TIME from 90 to 30 seconds, reap every 30 seconds

* Use a shared pool that returns preferred connection but re-purposes other ones when needed

* Fix wrong connection being returned on subsequent calls within the same thread

* Reduce mutex calls on flushes from 2 to 1 and add test for reaping
2019-07-02 00:34:38 +02:00
ThibG 154c8a0302 Increase DNS timeouts (#10238) 2019-03-11 13:27:57 +01:00
Eugen Rochko ab9308b7f7 Fix ThreadResolveWorker getting queued with invalid URLs (#9628) 2018-12-26 19:15:53 +01:00
Eugen Rochko 030ce53623 Fix TLS handshake timeout not being enforced (#9381)
Follow-up to #9329
2018-11-27 19:46:05 +01:00
Eugen Rochko 8c7ba82dc1 Fix nil error when no DNS addresses are found for host (#9379) 2018-11-27 18:13:36 +01:00
Eugen Rochko f3fee3cd27 Fix connect timeout not being enforced (#9329)
* Fix connect timeout not being enforced

The loop was catching the timeout exception that should stop execution, so the next IP would no longer be within a timed block, which led to requests taking much longer than 10 seconds.

* Use timeout on each IP attempt, but limit to 2 attempts

* Fix code style issue

* Do not break Request#perform if no block given

* Update method stub in spec for Request

* Move timeout inside the begin/rescue block

* Use Resolv::DNS with timeout of 1 to get IP addresses

* Update Request spec to stub Resolv::DNS instead of Addrinfo

* Fix Resolve::DNS stubs in Request spec
2018-11-22 20:12:04 +01:00
Eugen Rochko 0ebb6c1480 Revert connect timeout from 1s to 10s (#9319)
The failure rate in Sidekiq is too high
2018-11-21 17:00:56 +01:00
Eugen Rochko d78aed7a37 Reduce connect timeout limit and limit signature failures by source IP (#9236)
* Reduce connect timeout from 10s to 1s

* Limit failing signature verifications per source IP
2018-11-08 21:35:58 +01:00
Eugen Rochko b8e126ca0f Do not sign useless User-Agent or Accept-Encoding headers (#8533)
Fix #8080
2018-08-31 04:22:52 +02:00
Eugen Rochko 1d319c531e Add CLI task for rotating keys (#8466)
* If an Update is signed with known key, skip re-following procedure

Because it means the remote actor did *not* lose their database

* Add CLI method for rotating keys

    bin/tootctl accounts rotate [USERNAME]

Generates a new RSA key per account and sends out an Update activity
signed with the old key.

* Key rotation: Space out Update fan-outs every 5 minutes per 1000 accounts

* Skip suspended accounts in key rotation
2018-08-26 20:21:03 +02:00
MIYAGI Hikaru ac56fa3c22 Merge `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` into `ALLOW_ACCESS_TO_HIDDEN_SERVICE` (#7901)
If Mastodon accesses to the hidden service via transparent proxy, it's needed to avoid checking whether it's a private address, since `.onion` is resolved to a private address.
I was previously using the `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` to provide that function. However, I realized that using `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` is redundant, since this specification is always used with `ALLOW_ACCESS_TO_HIDDEN_SERVICE`. Therefore, I decided to integrate the setting of `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` into` ALLOW_ACCESS_TO_HIDDEN_SERVICE`.
2018-06-29 15:36:02 +02:00
MIYAGI Hikaru 7dbf83d6b8 User agent for WebFinger (#7531)
* User agent for WebFinger

* local_domain → web_domain

* 'http' is away accidentally...
2018-05-18 01:47:22 +02:00
Yamagishi Kazutoshi 59b48b923d Disable inflate gzip on head method (#7432) 2018-05-10 14:36:12 +02:00
Yamagishi Kazutoshi 1b42ea9f2c Support gzip encoding on Request (#7425) 2018-05-09 16:59:39 +02:00
Eugen Rochko 192746ec3e Guard against nil URLs in Request class (#7284)
Fix #7265
2018-05-02 15:44:22 +02:00
MIYAGI Hikaru 28808f638e HTTP proxy support for outgoing request, manage access to hidden service (#7134)
* Add support for HTTP client proxy

* Add access control for darknet

Supress error when access to darknet via transparent proxy

* Fix the codes pointed out

* Lint

* Fix an omission + lint

* any? -> include?

* Change detection method to regexp to avoid test fail
2018-04-25 02:14:49 +02:00
Akihiko Odaki acece7a2e6 Validate HTTP response length while receiving (#6891)
to_s method of HTTP::Response keeps blocking while it receives the whole
content, no matter how it is big. This means it may waste time to receive
unacceptably large files. It may also consume memory and disk in the
process. This solves the inefficency by checking response length while
receiving.
2018-03-26 14:02:10 +02:00
Akihiko Odaki 11c19f6cc9 Close http connection in perform method of Request class (#6889)
HTTP connections must be explicitly closed in many cases, and letting
perform method close connections makes its callers less redundant and
prevent them from forgetting to close connections.
2018-03-24 12:49:54 +01:00
Rey Tucker 4c91be94d3 request: in the event of failure, try other IPs (#6761) (#6813)
* request: in the event of failure, try other IPs (#6761)

In the case where a name has multiple A/AAAA records, we should
try subsequent records instead of immediately failing when we have a
failure on the first IP address.

This significantly improves delivery success when there are network
connectivity problems affecting only IPv4 or IPv6.

* fix method call style

* request_spec: adjust test case to use Addrinfo

* request: Request/open: move private addr check to within begin/rescue

* request_spec: add case to test failover, fix exception check

* Double Addrinfo.foreach so that it correctly yields instances
2018-03-20 09:06:08 +01:00
Akihiko Odaki a5a434a8f6 Raise Mastodon::HostValidationError when host for HTTP request is private (#6410) 2018-02-24 19:16:11 +01:00
Yamagishi Kazutoshi fba46b6072 Using double splat operator (#5859) 2017-12-06 11:41:57 +01:00
Daigo 3 Dango 87223e9426 Suppress backtrace from Request#perform (#5174) 2017-10-02 03:02:08 +02:00
Eugen Rochko 7143870551 Change max redirects followed to 2 (#5136)
I see no reason to allow more than that. Usually a redirect is
HTTP->HTTPS, then maybe URL structure changed, but more than that
is highly unlikely to be a legitimate use case.
2017-09-28 23:20:08 +02:00
unarist f7ae3f39f6 Include requested URL into the message on network errors (#4945) 2017-09-14 16:12:50 +02:00
Eugen Rochko 1b6ced5887 Add Digest header to requests with body, handle acct and URI keyId (#4565) 2017-08-09 23:54:14 +02:00
Eugen Rochko c5fa4aba91 HTTP signatures (#4146)
* Add Request class with HTTP signature generator

Spec: https://tools.ietf.org/html/draft-cavage-http-signatures-06

* Add HTTP signature verification concern

* Add test for SignatureVerification concern

* Add basic test for Request class

* Make PuSH subscribe/unsubscribe requests use new Request class

Accidentally fix lease_seconds not being set and sent properly, and
change the new minimum subscription duration to 1 day

* Make all PuSH workers use new Request class

* Make Salmon sender use new Request class

* Make FetchLinkService use new Request class

* Make FetchAtomService use the new Request class

* Make Remotable use the new Request class

* Make ResolveRemoteAccountService use the new Request class

* Add more tests

* Allow +-30 seconds window for signed request to remain valid

* Disable time window validation for signed requests, restore 7 days
as PuSH subscription duration (which was previous default due to a bug)
2017-07-14 20:41:49 +02:00