Commit Graph

119 Commits (8d4e705cb0d69c4a6b581b3dca482f7a2cc8b2de)

Author SHA1 Message Date
Eugen Rochko ca76d11434 Forward ActivityPub creates that reply to local statuses (#4709)
* Forward ActivityPub creates that reply to local statuses

* Fix test

* Fix wrong signers
2017-08-30 15:37:02 +02:00
Eugen Rochko 3135d20283 Serialize ActivityPub alternate link into OStatus deletes, handle it (#4730)
Requires moving Atom rendering from DistributionWorker (where
`stream_entry.status` is already nil) to inline (where
`stream_entry.status.destroyed?` is true) and distributing that.

Unfortunately, such XML renderings can no longer be easily chained
together into one payload of n items.
2017-08-29 16:11:05 +02:00
Eugen Rochko c423a3463b Forward ActivityPub deletes to followers of rebloggers (#4706) 2017-08-26 18:52:53 +02:00
Eugen Rochko 5147147da9 Add handling of Linked Data Signatures in payloads (#4687)
* Add handling of Linked Data Signatures in payloads

* Add a way to sign JSON, fix canonicalization of signature options

* Fix signatureValue encoding, send out signed JSON when distributing

* Add missing security context
2017-08-26 13:47:38 +02:00
masarakki d8f136cb8b authorize-follow-requests-after-unlocking (#4658) 2017-08-26 12:40:03 +02:00
Eugen Rochko cab7fa158a Add configuration to disable private status federation over PuSH (#4582) 2017-08-24 17:51:32 +02:00
Eugen Rochko 0268b334ae Periodically remove expired PuSH subscribers (#4654) 2017-08-21 22:56:33 +02:00
Eugen Rochko 1a19358b24 ActivityPub migration procedure (#4617)
* ActivityPub migration procedure

Once one account is detected as going from OStatus to ActivityPub,
invalidate WebFinger cache for other accounts from the same domain

* Unsubscribe from PuSH updates once we receive an ActivityPub payload

* Re-subscribe to PuSH unless already unsubscribed, regardless of protocol
2017-08-21 01:14:40 +02:00
Eugen Rochko aee1db8fa1 Improve ActivityPub/OStatus compatibility (#4632)
*Note: OStatus URIs are invalid for ActivityPub. But we have them for
as long as we want to keep old OStatus-sourced content and as long as
we remain OStatus-compatible.*

- In Announce handling, if object URI is not a URL, fallback to object URL
- Do not use specialized ThreadResolveWorker, rely on generalized handling
- When serializing notes, if parent's URI is not a URL, use parent's URL
2017-08-19 18:44:48 +02:00
Eugen Rochko 5516767c75 ActivityPub delivery (#4566)
* Deliver ActivityPub Like

* Deliver ActivityPub Undo-Like

* Deliver ActivityPub Create/Announce activities

* Deliver ActivityPub creates from mentions

* Deliver ActivityPub Block/Undo-Block

* Deliver ActivityPub Accept/Reject-Follow

* Deliver ActivityPub Undo-Follow

* Deliver ActivityPub Follow

* Deliver ActivityPub Delete activities

Incidentally fix #889

* Adjust BatchedRemoveStatusService for ActivityPub

* Add tests for ActivityPub workers

* Add tests for FollowService

* Add tests for FavouriteService, UnfollowService and PostStatusService

* Add tests for ReblogService, BlockService, UnblockService, ProcessMentionsService

* Add tests for AuthorizeFollowService, RejectFollowService, RemoveStatusService

* Add tests for BatchedRemoveStatusService

* Deliver updates to a local account to ActivityPub followers

* Minor adjustments
2017-08-13 00:44:41 +02:00
Eugen Rochko b33ac4a307 Only PuSH-resubscribe to OStatus accounts (#4583) 2017-08-12 02:54:54 +02:00
Eugen Rochko f18739fd60 Add ActivityPub inbox (#4216)
* Add ActivityPub inbox

* Handle ActivityPub deletes

* Handle ActivityPub creates

* Handle ActivityPub announces

* Stubs for handling all activities that need to be handled

* Add ActivityPub actor resolving

* Handle conversation URI passing in ActivityPub

* Handle content language in ActivityPub

* Send accept header when fetching actor, handle JSON parse errors

* Test for ActivityPub::FetchRemoteAccountService

* Handle public key and icon/image when embedded/as array/as resolvable URI

* Implement ActivityPub::FetchRemoteStatusService

* Add stubs for more interactions

* Undo activities implemented

* Handle out of order activities

* Hook up ActivityPub to ResolveRemoteAccountService, handle
Update Account activities

* Add fragment IDs to all transient activity serializers

* Add tests and fixes

* Add stubs for missing tests

* Add more tests

* Add more tests
2017-08-08 21:52:15 +02:00
Yamagishi Kazutoshi f95fca19d4 Change RuboCop rules to loose (#4464) 2017-07-31 15:19:13 +02:00
Eugen Rochko 64a6609d3e Fix guard clause in WebPushNotificationWorker (#4421) 2017-07-28 17:21:28 +02:00
Eugen Rochko a8631a9aac When PuSH subscribe attempts are exhausted, unsubscribe (#4422) 2017-07-28 17:21:14 +02:00
Clworld 0c86b36cac Add callback_url/acct information for Sidekiq PuSH workers Exception. (#4281)
* Add destination informations to exception on SubscribeWorker and DeliveryWorker.

* Simplify delivery error message.

* Prevent changing Exception type...

* fix typo.
2017-07-27 00:38:20 +02:00
Sorin Davidoi eb30328beb fix(web_push_notification_worker): Guard against deleted notifications (#4379) 2017-07-26 16:14:39 +02:00
Eugen Rochko 97dda48f70 Move clean up of unconfirmed users to sidekiq-scheduler (#4336)
* Move clean up of unconfirmed users to sidekiq-scheduler

* mastodon:daily is now deprecated
2017-07-24 16:17:55 +02:00
Eugen Rochko 75ab767791 Fix webfinger retries (#4275)
* Do not raise unretryable exceptions in ResolveRemoteAccountService

* Removed fatal exceptions from ResolveRemoteAccountService

Exceptions that cannot be retried should not be raised. New exception
class for those that can be retried (Mastodon::UnexpectedResponseError)
2017-07-20 01:59:07 +02:00
Eugen Rochko 3852032953 Correct OStatus inflection (Ostatus -> OStatus) (#4255) 2017-07-19 01:37:26 +02:00
Akihiko Odaki b0f97d9a87 Introduce Ostatus name space (#4164)
* Wrap methods of ProcessFeedService::ProcessEntry in classes

This is a change same with 002ed7dc62, except
that it has the following changes:

* Revert irrelevant change in find_or_create_conversation
* Fix error handling for RemoteActivity

* Introduce Ostatus name space
2017-07-18 16:39:47 +02:00
Sorin Davidoi 3d4ae604b8 fix(push-subscriptions): Refactor how Sidekiq jobs are handled (#4226) 2017-07-18 16:25:40 +02:00
Akihiko Odaki 6eff4496e3 Improve a query in WebPushNotificationWorker (#4234) 2017-07-17 11:03:48 +02:00
Akihiko Odaki 71a02eb2d9 Use update method to update session in WebPushNotificationWorker (#4235) 2017-07-17 11:03:42 +02:00
ThibG 9c1791c546 Optimize uri normalization (#4212)
* Add dependency on idn-ruby to speed up URI normalization

* Use normalized_host instead of normalize.host when applicable

When we are only interested in the normalized host, calling normalized_host
avoids normalizing the other components of the URI as well as creating a
new object
2017-07-15 17:24:35 +02:00
Eugen Rochko c6e46f7421 Fix #2672 - Connect signed PuSH subscription requests to instance domain (#4205)
* Fix #2672 - Connect signed PuSH subscription requests to instance domain

Resolves #2739

* Fix return of locate_subscription

* Fix tests
2017-07-14 23:01:20 +02:00
Eugen Rochko c5fa4aba91 HTTP signatures (#4146)
* Add Request class with HTTP signature generator

Spec: https://tools.ietf.org/html/draft-cavage-http-signatures-06

* Add HTTP signature verification concern

* Add test for SignatureVerification concern

* Add basic test for Request class

* Make PuSH subscribe/unsubscribe requests use new Request class

Accidentally fix lease_seconds not being set and sent properly, and
change the new minimum subscription duration to 1 day

* Make all PuSH workers use new Request class

* Make Salmon sender use new Request class

* Make FetchLinkService use new Request class

* Make FetchAtomService use the new Request class

* Make Remotable use the new Request class

* Make ResolveRemoteAccountService use the new Request class

* Add more tests

* Allow +-30 seconds window for signed request to remain valid

* Disable time window validation for signed requests, restore 7 days
as PuSH subscription duration (which was previous default due to a bug)
2017-07-14 20:41:49 +02:00
Sorin Davidoi ecab38fd66 Web Push Notifications (#3243)
* feat: Register push subscription

* feat: Notify when mentioned

* feat: Boost, favourite, reply, follow, follow request

* feat: Notification interaction

* feat: Handle change of public key

* feat: Unsubscribe if things go wrong

* feat: Do not send normal notifications if push is enabled

* feat: Focus client if open

* refactor: Move push logic to WebPushSubscription

* feat: Better title and body

* feat: Localize messages

* chore: Fix lint errors

* feat: Settings

* refactor: Lazy load

* fix: Check if push settings exist

* feat: Device-based preferences

* refactor: Simplify logic

* refactor: Pull request feedback

* refactor: Pull request feedback

* refactor: Create /api/web/push_subscriptions endpoint

* feat: Spec PushSubscriptionController

* refactor: WebPushSubscription => Web::PushSubscription

* feat: Spec Web::PushSubscription

* feat: Display first media attachment

* feat: Support direction

* fix: Stuff broken while rebasing

* refactor: Integration with session activations

* refactor: Cleanup

* refactor: Simplify implementation

* feat: Set VAPID keys via environment

* chore: Comments

* fix: Crash when no alerts

* fix: Set VAPID keys in testing environment

* fix: Follow link

* feat: Notification actions

* fix: Delete previous subscription

* chore: Temporary logs

* refactor: Move migration to a later date

* fix: Fetch the correct session activation and misc bugs

* refactor: Move migration to a later date

* fix: Remove follow request (no notifications)

* feat: Send administrator contact to push service

* feat: Set time-to-live

* fix: Do not show sensitive images

* fix: Reducer crash in error handling

* feat: Add badge

* chore: Fix lint error

* fix: Checkbox label overlap

* fix: Check for payload support

* fix: Rename action "type" (crash in latest Chrome)

* feat: Action to expand notification

* fix: Lint errors

* fix: Unescape notification body

* fix: Do not allow boosting if the status is hidden

* feat: Add VAPID keys to the production sample environment

* fix: Strip HTML tags from status

* refactor: Better error messages

* refactor: Handle browser not implementing the VAPID protocol (Samsung Internet)

* fix: Error when target_status is nil

* fix: Handle lack of image

* fix: Delete reference to invalid subscriptions

* feat: Better error handling

* fix: Unescape HTML characters after tags are striped

* refactor: Simpify code

* fix: Modify to work with #4091

* Sort strings alphabetically

* i18n: Updated Polish translation

it annoys me that it's not fully localized :P

* refactor: Use current_session in PushSubscriptionController

* fix: Rebase mistake

* fix: Set cacheName to mastodon

* refactor: Pull request feedback

* refactor: Remove logging statements

* chore(yarn): Fix conflicts with master

* chore(yarn): Copy latest from master

* chore(yarn): Readd offline-plugin

* refactor: Use save! and update!

* refactor: Send notifications async

* fix: Allow retry when push fails

* fix: Save track for failed pushes

* fix: Minify sw.js

* fix: Remove account_id from fabricator
2017-07-13 22:15:32 +02:00
Eugen Rochko af4835f176 Limit total subscribe retries to 10, but space them out more (#4142)
Since there is little point in retrying so often when a service is down
or does not exist anymore. Subscriptions are renewed 1 day before they
should expire, so retrying in 30 minutes, then 2 hours, then 12 hours
is fine. If even after that, the remote server does not work, there is
little sense in retrying more often than once a day

Also, uniqueness of the job should ensure that failed retries will
not result in multiple retries for the same endpoint when the next
resubscription cycle comes
2017-07-10 18:04:23 +02:00
Eugen Rochko 20e15ecfb3 Refactor JSON templates to be generated with ActiveModelSerializers instead of Rabl (#4090) 2017-07-07 04:02:06 +02:00
Eugen Rochko ad531f90b1 Bind web UI access tokens to sessions (#3940)
* Add overview of active sessions

* Better display of browser/platform name

* Improve how browser information is stored and displayed for sessions overview

* Fix test

* Fix #2347 - Bind web UI access token to session

When you logout, session also destroys the access token, so it's no longer
valid. If access token is destroyed some other way, the session is also
destroyed, requiring a re-login.

Fix #1681 - Add scheduler to remove revoked access tokens and grants

* Fix test
2017-06-25 23:51:32 +02:00
Akihiko Odaki (@fn_aki@pawoo.net) 561ca10d86 Some minor change and spec for Account (#3813)
* Introduce domains method to Account relation

Account had followers_domains method, which was excessively specific.
Let relation of Account have domains method instead.

* Move follow_mapping in Account to AccountInteractions

* Introduce shared examples for AccountAvatar inclusion

* Cover Account more
2017-06-22 18:34:27 +02:00
Eugen Rochko 8bed91d94c Rename FollowRemoteAccountService to ResolveRemoteAccountService (#3847)
Rename Activitypub to ActivityPub
2017-06-19 01:51:04 +02:00
Eugen Rochko 6f8f401ea1 Batched remove status service (#3735)
* Make Pubsubhubbub::DistributionWorker handle both single stream entry
arguments, as well as arrays of stream entries

* Add BatchedRemoveStatusService, make SuspendAccountService use it

* Improve method names

* Add test

* Add more tests

* Use PuSH payloads of 100 to have a clear mapping of
1000 input statuses -> 10 PuSH payloads

It was nice while it lasted
2017-06-14 18:01:35 +02:00
Eugen Rochko 1c7e2ddd65 Account deletion (#3728)
* Add form for account deletion

* If avatar or header are gone from source, remove them

* Add option to have SuspendAccountService remove user record, add tests

* Exclude suspended accounts from search
2017-06-14 18:01:27 +02:00
takayamaki aa6740c21b change sidekiq queueing to bulk push (#3536) 2017-06-04 00:11:15 +02:00
Matt Jankowski 0eb8b00cc9 Specs for cleanup workers (#3235)
* Add spec files for feed and media cleanup workers

* Add coverage for feed and media cleanup schedulers

* Clean up feed and media cleanup workers
2017-05-22 19:36:21 +02:00
Eugen Rochko b51beb6486 Replace mastodon:media:clear and mastodon:feeds:clear rake tasks with (#3180)
sidekiq-scheduler jobs

Resolves #2495
2017-05-20 19:42:58 +02:00
Matt Jankowski c8f8629c7e Add spec coverage for regeneration worker (#3143) 2017-05-19 16:55:09 +02:00
Matt Jankowski 4423ed3557 Specs for precompute feed service (#3142)
* Add spec for precompute feed service

* Refactor PrecomputeFeedService

* spec wip
2017-05-19 16:21:52 +02:00
Matt Jankowski b3f0e6f7b9 After remote follow worker specs (#3098)
* Add specs for the after remote follow workers

* Refactor the after remote follow workers
2017-05-18 21:10:41 +02:00
Eugen Rochko 83b444ddab Do not cancel PuSH subscriptions after encountering "permanent" error… (#3046)
* Do not cancel PuSH subscriptions after encountering "permanent" error response

After talking with MMN about it, turns out some servers/php setups do
return 4xx errors while rebooting, so this anti-feature that was meant
to take load off of the hub is doing more harm than good in terms of
breaking subscriptions

* Update delivery_worker.rb
2017-05-14 03:22:48 +02:00
Matt Jankowski dc59643e73 Spec coverage and refactor for pubsub/delivery worker (#3021)
* Framework for delivery worker spec

* Refactor of pubsub delivery worker
2017-05-12 20:35:36 +02:00
Matt Jankowski 60fa69399c Pubsub confirmation worker spec (#2974)
* Add specs for valid challenge response from pubsub confirmation worker

* Refactor the pubsub confirmation worker
2017-05-11 00:19:37 +02:00
Matt Jankowski cd8fc8031f Spec feed insert worker (#2965)
* Spec for feed insert worker when missing records

* more specs!

* Refactor feed insert worker
2017-05-10 20:32:05 +02:00
Matt Jankowski cd3630e3d1 Spec coverage and refactor of digest mailer worker (#2961) 2017-05-10 15:45:43 +02:00
Eugen Rochko 221898cc90 Add spec for Pubsubhubbub::DistributionWorker. PuSH-deliver public items (#2954)
to all subscribers. IDN-normalize callback URLs for subscriptions on insert.
2017-05-10 02:55:43 +02:00
haoyayoi a5ad348182 add content-type in pubsubhubhub request header (#2943)
* add content-type in pubsubhubhub request header

* fix type
2017-05-09 16:34:47 +02:00
Eugen Rochko 78b890cc81 Fix Scheduler::SubscriptionsScheduler (#2834)
* Fix Scheduler::SubscriptionsScheduler, add worker test for it

* Change production log level of Sidekiq to "warn" instead of "info"
2017-05-06 13:05:03 +02:00
Eugen Rochko c32e312061 More robust PuSH subscription refreshes (#2799)
* Fix #2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron

Fix an issue where / in domain would raise exception in TagManager#normalize_domain

PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through

Fix #2702 - Correct user agent header on outgoing http requests

* Add test for SubscribeService

* Extract #expiring_accounts into method

* Make mastodon:push:refresh no-op

* Queues are now defined in sidekiq.yml

* Queues are now in sidekiq.yml
2017-05-05 02:23:01 +02:00