mastodon/app/models
ThibG 694a5e33d5 Add follower synchronization mechanism (#14510)
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
..
admin Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
concerns Add follower synchronization mechanism (#14510) 2020-10-21 18:04:09 +02:00
form Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
web Bump doorkeeper from 5.3.3 to 5.4.0 (#13733) 2020-05-12 15:25:33 +02:00
account.rb Add follower synchronization mechanism (#14510) 2020-10-21 18:04:09 +02:00
account_alias.rb Remove dependency on goldfinger gem (#14919) 2020-10-08 00:34:57 +02:00
account_conversation.rb allow pagination by min_id and max_id (#14776) 2020-09-12 17:09:49 +02:00
account_deletion_request.rb Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
account_domain_block.rb Fix crash when saving invalid domain name (#11528) 2019-08-08 23:04:19 +02:00
account_filter.rb Add sorting by username, creation and last activity in moderation view (#13076) 2020-03-08 15:39:13 +01:00
account_identity_proof.rb Fix Keybase verification using wrong domain for remote accounts (#10547) 2019-04-10 20:28:43 +02:00
account_migration.rb Remove dependency on goldfinger gem (#14919) 2020-10-08 00:34:57 +02:00
account_moderation_note.rb Update dependencies for Ruby (2018-04-23) (#7237) 2018-04-23 11:29:17 +02:00
account_note.rb Add user notes on accounts (#14148) 2020-06-30 19:19:50 +02:00
account_pin.rb Add API endpoint to list featured accounts (fixes #8315) (#8317) 2018-08-20 18:46:04 +02:00
account_stat.rb Fix account counters being overwritten by parallel writes (#12045) 2019-10-02 04:59:37 +02:00
account_tag_stat.rb Add profile directory (#9427) 2018-12-06 17:36:11 +01:00
account_warning.rb Add moderation warnings (#9519) 2018-12-22 20:02:09 +01:00
account_warning_preset.rb Add titles to warning presets in admin UI (#13252) 2020-03-12 17:57:59 +01:00
admin.rb Add logging of admin actions (#5757) 2017-11-24 02:05:53 +01:00
announcement.rb Change delivery failure tracking to work with hostnames instead of URLs (#13437) 2020-04-15 20:33:24 +02:00
announcement_filter.rb Add announcements (#12662) 2020-01-23 22:00:13 +01:00
announcement_mute.rb Add announcements (#12662) 2020-01-23 22:00:13 +01:00
announcement_reaction.rb Add announcements (#12662) 2020-01-23 22:00:13 +01:00
application_record.rb Fix records not being indexed sometimes (#12024) 2019-10-01 01:19:11 +02:00
backup.rb Add announcements (#12662) 2020-01-23 22:00:13 +01:00
block.rb Store URIs of follows, follow requests and blocks for ActivityPub (#7160) 2018-05-04 21:14:34 +02:00
bookmark.rb Add announcements (#12662) 2020-01-23 22:00:13 +01:00
context.rb Refactor JSON templates to be generated with ActiveModelSerializers instead of Rabl (#4090) 2017-07-07 04:02:06 +02:00
conversation.rb Revert "Remove conversation URI (#11423)" (#11424) 2019-07-28 17:47:37 +02:00
conversation_mute.rb Update dependencies for Ruby (2018-04-23) (#7237) 2018-04-23 11:29:17 +02:00
custom_emoji.rb Add separate cache directory for non-local uploads (#12821) 2020-04-26 23:29:08 +02:00
custom_emoji_category.rb Add batch actions and categories to admin UI for custom emojis (#11793) 2019-09-09 22:44:17 +02:00
custom_emoji_filter.rb Various fixes and improvements (#12878) 2020-01-20 15:55:03 +01:00
custom_filter.rb Add “account timeline” filter category (#12918) 2020-01-23 21:32:00 +01:00
device.rb Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
domain_allow.rb Fix crash when saving invalid domain name (#11528) 2019-08-08 23:04:19 +02:00
domain_block.rb Fix admin page crashing when trying to block an invalid domain name (#13884) 2020-06-01 03:47:20 +02:00
email_domain_block.rb Add option to include resolved DNS records when blacklisting e-mail domains in admin UI (#13254) 2020-03-12 22:35:20 +01:00
encrypted_message.rb Change Redis#exists calls to Redis#exists? to avoid deprecation warning (#14191) 2020-07-01 19:05:21 +02:00
export.rb Export and import show_reblogs together with following list (#10495) 2019-04-08 07:28:27 +02:00
favourite.rb Fix records not being indexed sometimes (#12024) 2019-10-01 01:19:11 +02:00
featured_tag.rb Fix featured tags not finding the right tag on save (#11504) 2019-08-07 10:00:58 +02:00
feed.rb allow pagination by min_id and max_id (#14776) 2020-09-12 17:09:49 +02:00
follow.rb Add follower synchronization mechanism (#14510) 2020-10-21 18:04:09 +02:00
follow_request.rb Add option to be notified when a followed user posts (#13546) 2020-09-18 17:26:45 +02:00
home_feed.rb Change Redis#exists calls to Redis#exists? to avoid deprecation warning (#14191) 2020-07-01 19:05:21 +02:00
identity.rb Change identities id column to a bigint (#9371) 2018-11-27 13:56:25 +01:00
import.rb Fix csv upload (#13835) 2020-05-24 09:15:23 +02:00
instance.rb Add whitelist mode (#11291) 2019-07-30 11:10:46 +02:00
instance_filter.rb Various fixes and improvements (#12878) 2020-01-20 15:55:03 +01:00
invite.rb Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
invite_filter.rb Various fixes and improvements (#12878) 2020-01-20 15:55:03 +01:00
ip_block.rb Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
list.rb Add configuration option to filter replies in lists (#9205) 2020-09-01 13:31:28 +02:00
list_account.rb Add abilityto add oneself to lists (#12271) 2019-11-04 13:02:01 +01:00
list_feed.rb Lists (#5703) 2017-11-18 00:16:48 +01:00
marker.rb Add timeline read markers API (#11762) 2019-09-06 13:55:51 +02:00
media_attachment.rb Fix videos with near-60 fps being rejected (#14684) 2020-08-30 01:54:30 +02:00
mention.rb Improve support for aspects/circles (#8950) 2018-10-17 17:13:04 +02:00
message_franking.rb Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
mute.rb Add duration parameter to muting. (#13831) 2020-10-13 01:01:14 +02:00
notification.rb Add option to be notified when a followed user posts (#13546) 2020-09-18 17:26:45 +02:00
one_time_key.rb Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
poll.rb Only normalize local polls (#12515) 2019-12-01 17:24:33 +01:00
poll_vote.rb Add optimistic lock to avoid race conditions when handling votes (#10196) 2019-03-06 19:53:57 +01:00
preview_card.rb Bump rubocop from 0.86.0 to 0.88.0 (#14412) 2020-09-01 03:04:00 +02:00
public_feed.rb Refactor how public and tag timelines are queried (#14728) 2020-09-07 11:02:04 +02:00
relationship_filter.rb Fix followings list order | Issue #13538 (#13676) 2020-05-08 20:17:16 +02:00
relay.rb Fix enable/disable relay failures (#13535) 2020-04-23 22:04:18 +02:00
remote_follow.rb Remove dependency on goldfinger gem (#14919) 2020-10-08 00:34:57 +02:00
report.rb Add rate limit for reporting (#13390) 2020-04-05 14:40:08 +02:00
report_filter.rb Various fixes and improvements (#12878) 2020-01-20 15:55:03 +01:00
report_note.rb Fix scope latest of ReportNote (#9630) 2018-12-26 06:38:59 +01:00
scheduled_status.rb Fix deleting a scheduled status immediately deleting media attachments (#9728) 2019-01-06 16:38:40 +01:00
search.rb Refactor JSON templates to be generated with ActiveModelSerializers instead of Rabl (#4090) 2017-07-07 04:02:06 +02:00
session_activation.rb Update dependencies for Ruby (2018-04-23) (#7237) 2018-04-23 11:29:17 +02:00
setting.rb Remove intermediary arrays when creating hash maps from results (#9291) 2018-11-16 15:02:18 +01:00
site_upload.rb Improvements to image upload validation and creation (#10431) 2019-04-01 07:30:46 +02:00
status.rb Fix a slow query for TagFeed (#14861) 2020-09-23 16:01:54 +02:00
status_pin.rb Update dependencies for Ruby (2018-04-23) (#7237) 2018-04-23 11:29:17 +02:00
status_stat.rb Reset status cache when status_stat or media_attachment updates (#9119) 2018-10-28 06:42:34 +01:00
system_key.rb Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
tag.rb Changed tag most_used to recently_used (#14760) 2020-09-07 17:47:41 +02:00
tag_feed.rb Fix a slow query for TagFeed (#14861) 2020-09-23 16:01:54 +02:00
tag_filter.rb Various fixes and improvements (#12878) 2020-01-20 15:55:03 +01:00
tombstone.rb Record deleted(by mod) status to prevent re-appear (#10732) 2019-05-09 22:03:02 +02:00
trending_tags.rb Fix admin setting to auto-approve hashtags not affecting query (#12130) 2019-10-10 02:22:04 +02:00
unavailable_domain.rb Change delivery failure tracking to work with hostnames instead of URLs (#13437) 2020-04-15 20:33:24 +02:00
user.rb Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
user_invite_request.rb Add "why do you want to join" field to invite requests (#10524) 2019-04-09 23:06:30 +09:00
web.rb Add extended about page stub 2017-01-13 03:24:41 +01:00
webauthn_credential.rb Fix validates :sign_count of WebauthnCredential (#14806) 2020-09-16 20:16:46 +02:00