Commit Graph

91 Commits (fdd9d8eecf2fbdd9164542f083c5016a5061957e)

Author SHA1 Message Date
Eugen Rochko af4835f176 Limit total subscribe retries to 10, but space them out more (#4142)
Since there is little point in retrying so often when a service is down
or does not exist anymore. Subscriptions are renewed 1 day before they
should expire, so retrying in 30 minutes, then 2 hours, then 12 hours
is fine. If even after that, the remote server does not work, there is
little sense in retrying more often than once a day

Also, uniqueness of the job should ensure that failed retries will
not result in multiple retries for the same endpoint when the next
resubscription cycle comes
2017-07-10 18:04:23 +02:00
Eugen Rochko 20e15ecfb3 Refactor JSON templates to be generated with ActiveModelSerializers instead of Rabl (#4090) 2017-07-07 04:02:06 +02:00
Eugen Rochko ad531f90b1 Bind web UI access tokens to sessions (#3940)
* Add overview of active sessions

* Better display of browser/platform name

* Improve how browser information is stored and displayed for sessions overview

* Fix test

* Fix #2347 - Bind web UI access token to session

When you logout, session also destroys the access token, so it's no longer
valid. If access token is destroyed some other way, the session is also
destroyed, requiring a re-login.

Fix #1681 - Add scheduler to remove revoked access tokens and grants

* Fix test
2017-06-25 23:51:32 +02:00
Akihiko Odaki (@fn_aki@pawoo.net) 561ca10d86 Some minor change and spec for Account (#3813)
* Introduce domains method to Account relation

Account had followers_domains method, which was excessively specific.
Let relation of Account have domains method instead.

* Move follow_mapping in Account to AccountInteractions

* Introduce shared examples for AccountAvatar inclusion

* Cover Account more
2017-06-22 18:34:27 +02:00
Eugen Rochko 8bed91d94c Rename FollowRemoteAccountService to ResolveRemoteAccountService (#3847)
Rename Activitypub to ActivityPub
2017-06-19 01:51:04 +02:00
Eugen Rochko 6f8f401ea1 Batched remove status service (#3735)
* Make Pubsubhubbub::DistributionWorker handle both single stream entry
arguments, as well as arrays of stream entries

* Add BatchedRemoveStatusService, make SuspendAccountService use it

* Improve method names

* Add test

* Add more tests

* Use PuSH payloads of 100 to have a clear mapping of
1000 input statuses -> 10 PuSH payloads

It was nice while it lasted
2017-06-14 18:01:35 +02:00
Eugen Rochko 1c7e2ddd65 Account deletion (#3728)
* Add form for account deletion

* If avatar or header are gone from source, remove them

* Add option to have SuspendAccountService remove user record, add tests

* Exclude suspended accounts from search
2017-06-14 18:01:27 +02:00
takayamaki aa6740c21b change sidekiq queueing to bulk push (#3536) 2017-06-04 00:11:15 +02:00
Matt Jankowski 0eb8b00cc9 Specs for cleanup workers (#3235)
* Add spec files for feed and media cleanup workers

* Add coverage for feed and media cleanup schedulers

* Clean up feed and media cleanup workers
2017-05-22 19:36:21 +02:00
Eugen Rochko b51beb6486 Replace mastodon:media:clear and mastodon:feeds:clear rake tasks with (#3180)
sidekiq-scheduler jobs

Resolves #2495
2017-05-20 19:42:58 +02:00
Matt Jankowski c8f8629c7e Add spec coverage for regeneration worker (#3143) 2017-05-19 16:55:09 +02:00
Matt Jankowski 4423ed3557 Specs for precompute feed service (#3142)
* Add spec for precompute feed service

* Refactor PrecomputeFeedService

* spec wip
2017-05-19 16:21:52 +02:00
Matt Jankowski b3f0e6f7b9 After remote follow worker specs (#3098)
* Add specs for the after remote follow workers

* Refactor the after remote follow workers
2017-05-18 21:10:41 +02:00
Eugen Rochko 83b444ddab Do not cancel PuSH subscriptions after encountering "permanent" error… (#3046)
* Do not cancel PuSH subscriptions after encountering "permanent" error response

After talking with MMN about it, turns out some servers/php setups do
return 4xx errors while rebooting, so this anti-feature that was meant
to take load off of the hub is doing more harm than good in terms of
breaking subscriptions

* Update delivery_worker.rb
2017-05-14 03:22:48 +02:00
Matt Jankowski dc59643e73 Spec coverage and refactor for pubsub/delivery worker (#3021)
* Framework for delivery worker spec

* Refactor of pubsub delivery worker
2017-05-12 20:35:36 +02:00
Matt Jankowski 60fa69399c Pubsub confirmation worker spec (#2974)
* Add specs for valid challenge response from pubsub confirmation worker

* Refactor the pubsub confirmation worker
2017-05-11 00:19:37 +02:00
Matt Jankowski cd8fc8031f Spec feed insert worker (#2965)
* Spec for feed insert worker when missing records

* more specs!

* Refactor feed insert worker
2017-05-10 20:32:05 +02:00
Matt Jankowski cd3630e3d1 Spec coverage and refactor of digest mailer worker (#2961) 2017-05-10 15:45:43 +02:00
Eugen Rochko 221898cc90 Add spec for Pubsubhubbub::DistributionWorker. PuSH-deliver public items (#2954)
to all subscribers. IDN-normalize callback URLs for subscriptions on insert.
2017-05-10 02:55:43 +02:00
haoyayoi a5ad348182 add content-type in pubsubhubhub request header (#2943)
* add content-type in pubsubhubhub request header

* fix type
2017-05-09 16:34:47 +02:00
Eugen Rochko 78b890cc81 Fix Scheduler::SubscriptionsScheduler (#2834)
* Fix Scheduler::SubscriptionsScheduler, add worker test for it

* Change production log level of Sidekiq to "warn" instead of "info"
2017-05-06 13:05:03 +02:00
Eugen Rochko c32e312061 More robust PuSH subscription refreshes (#2799)
* Fix #2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron

Fix an issue where / in domain would raise exception in TagManager#normalize_domain

PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through

Fix #2702 - Correct user agent header on outgoing http requests

* Add test for SubscribeService

* Extract #expiring_accounts into method

* Make mastodon:push:refresh no-op

* Queues are now defined in sidekiq.yml

* Queues are now in sidekiq.yml
2017-05-05 02:23:01 +02:00
Eugen 87f7a3922c Punycode URI normalization (#2370)
* Fix #2119 - Whenever about to send a HTTP request, normalize the URI

* Add test for IDN request in FetchLinkCardService

* Perform IDN normalization on domains before they are stored in the DB
2017-04-25 02:47:31 +02:00
Eugen f89e26bdd5 Followers-only post federation (#2111)
* Make private toots get PuSHed to subscription URLs that belong to domains where you have approved followers

* Authorized followers controller, stub for bulk action

* Soft block in the background

* Add simple test for new controller

* Rename Settings::FollowersController to Settings::FollowerDomainsController, paginate results,
rename "private" post setting to "followers-only", fix pagination style, improve post privacy
preferences style, improve warning style

* Extract compose form warnings into own container, show warning when posting to followers-only with unlocked account
2017-04-24 00:38:37 +02:00
Matt Jankowski d1ebb63c54 Quick best practice cleanup of views/helpers (#1546)
* Remove trailing whitespace

* Use query methods instead of explicit .blank? checks
2017-04-12 18:24:18 +02:00
Matt Jankowski bf7306636e Allow import/export of mutes list (#1541)
* Allow export of mutes list

* Allow importing of mutes list

* Refactor to use Settings::Exports::BaseController and DRY up exports code
2017-04-12 18:20:44 +02:00
Matt Jankowski db979ebcee Imports controller errors (#1553)
* Add spec for settings/imports controller

* Add failing spec for settings/imports#create

* Fix broken imports

* Refactor ImportWorker
2017-04-11 21:40:14 +02:00
Jantso Porali 4def9924d7 Fix for Import issue from URL (#1416) 2017-04-10 12:49:34 +02:00
Eugen 697e4da6cb Fix #1141, fix #1126 - Avatar/profile info fetching (#1215)
* Fix #1141, fix #1126 - Work through UpdateRemoteProfileService for both <feed> and <entry> top-level tags

* Improve code quality, remove line unrelated to fix
2017-04-08 13:26:03 +02:00
Eugen 24c77e57b2 Rewrite Atom generation from stream entries to use Ox instead of Nokogiri (#1124)
* Rewrite Atom generation from stream entries to use Ox instead of Nokogiri::Builder

StreamEntry is now limited to only statuses, which allows some optimization. Removed
extra queries on AccountsController#show. AtomSerializer instead of AtomBuilderHelper
used in AccountsController#show, StreamEntriesController#show, StreamEntryRenderer
and PubSubHubbub::DistributionWorker

PubSubHubbub::DistributionWorker moves n+1 DomainBlock query to PubSubHubbub::DeliveryWorker
instead.

All Salmon slaps that aren't based on StreamEntry still use AtomBuilderHelper and Nokogiri

* All Salmon slaps now use Ox instead of Nokogiri. No touch from status on account
2017-04-07 05:56:56 +02:00
blackle 70f485a9e7 Catch more errors in process_follows so it doesn't fail 2017-04-05 20:04:13 -04:00
Eugen Rochko cd68e54a7d Split SalmonWorker into smaller parts, move profile updating into another job 2017-04-05 21:43:10 +02:00
Eugen Rochko 5189ce9cd1 Replace calls to FeedManager#inline_render and #broadcast 2017-04-05 19:45:18 +02:00
Eugen 54329947ab Fix wrong variable used in publish channel 2017-04-05 18:58:32 +02:00
Eugen b9ddfff8f6 Add proper message to PushUpdateWorker, use redis directly 2017-04-05 18:48:41 +02:00
Eugen d67e1fe44d Merge branch 'master' into broadcast-to-worker 2017-04-05 18:44:33 +02:00
Eugen Rochko 8f38353c4d Only render public payload once in FanOutOnWrite 2017-04-05 14:26:17 +02:00
Kurtis Rainbolt-Greene 192f855952 We're going to want these nice helper methods, lets share them with a parent class that matches Rails 5 practices (application level abstraction) 2017-04-04 21:14:37 -07:00
Kurtis Rainbolt-Greene bea03b4553 Moving in the inline render 2017-04-04 20:51:18 -07:00
Kurtis Rainbolt-Greene dd66bce930 I don't actually think we need that. 2017-04-04 20:48:22 -07:00
Kurtis Rainbolt-Greene 7d887cea4b Moving the queue_at into the worker 2017-04-04 20:39:14 -07:00
Kurtis Rainbolt-Greene 4f2985cddc Replacing the broadcast method with the one defined in the feed manager 2017-04-04 20:36:03 -07:00
Kurtis Rainbolt-Greene de1bb4bd56 By pushing this into a worker we can reduce the amount of time the feed manager using workers eat up a connection 2017-04-04 20:23:40 -07:00
Eugen Rochko d2b7daf6bb Remove PuSH subscriptions when delivery is answered with a 4xx error 2017-04-05 02:34:33 +02:00
Eugen Rochko fa2a1af90a Spawn FeedInsertWorker to deliver status into personal feed 2017-04-04 19:21:37 +02:00
Eugen Rochko e60c473917 Reduce number of items in feeds, optimize regeneration worker slightly,
make regeneration worker unique, (only schedule/execute once at a time)
2017-04-04 13:58:34 +02:00
Eugen Rochko 1484df2db5 Move OStatus processing back into default queue 2017-04-04 13:02:49 +02:00
Eugen Rochko 7a56845060 Only call regeneration worker after first login after a 14 day break 2017-04-04 02:00:10 +02:00
Eugen Rochko 8eca42b265 Separate background jobs into different queues. ATTENTION: new queue "pull"
must be added to the Sidekiq invokation in your systemd file

The pull queue will handle link crawling, thread resolving, and OStatus
processing. Such tasks are more likely to hang for a longer time (due to
network requests) so it is more sensible to not make the "in-house" tasks
wait for them.
2017-04-04 00:53:20 +02:00
Eugen Rochko 6ebe31e716 Make default admin UI page reports. Add admin UI for creating a domain block 2017-04-03 18:55:06 +02:00