Commit Graph

1256 Commits (152f5c7983add9e5e2f56358142d4fcbc0443d39)

Author SHA1 Message Date
Claire f1a6f9062e Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/api/v1/statuses_controller.rb`:
  Upstream moved things around in a place where glitch-soc had support for
  an extra parameter (`content_type`).
  Follow upstream but reintroduce `content_type`.
2022-02-10 19:09:27 +01:00
Claire 63854bee6c
Fix poll votes not being properly reset on poll change (#17498)
* Fix poll votes not being properly reset on poll change

* Fix and add tests

* Fix poll update handling when the number of options changes
2022-02-10 14:26:54 +01:00
Eugen Rochko 1bfcb75105
Fix outdated iso-639 reference in update status service (#17496) 2022-02-10 03:09:44 +01:00
Eugen Rochko 63002cde03
Add editing for published statuses (#17320)
* Add editing for published statuses

* Fix change of multiple-choice boolean in poll not resetting votes

* Remove the ability to update existing media attachments for now
2022-02-10 00:15:30 +01:00
Eugen Rochko 2f8159baad
Add `category` and `rule_ids` params to `POST /api/v1/reports` (#17492) 2022-02-10 00:10:16 +01:00
Claire 692963d43b Merge branch 'main' into glitch-soc/merge-upstream 2022-02-08 18:23:53 +01:00
Eugen Rochko b6d7726ecb
Remove language detection through cld3 (#17478)
* Remove language detection through cld3

* Update app/helpers/languages_helper.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-08 02:41:17 +01:00
Claire c8b1e72a4f
Fix compacted JSON-LD possibly causing compatibility issues on forwarding (#17428) 2022-02-03 14:09:04 +01:00
Claire 948235592a
Fix response_to_recipient? CTE (#17427) 2022-02-03 14:07:43 +01:00
Claire d1ecc323e7
Compact JSON-LD signed incoming activities (#17426)
Co-authored-by: Puck Meerburg <puck@puck.moe>
2022-02-03 14:07:29 +01:00
Claire a3e0dacf5c Fix response_to_recipient? CTE 2022-02-02 19:55:57 +01:00
Claire 7b969436a0 Fix compacted JSON-LD possibly causing compatibility issues on forwarding 2022-02-02 19:55:57 +01:00
Puck Meerburg 63da32468c Compact JSON-LD signed incoming activities 2022-02-02 16:13:11 +01:00
Claire 94a39f6b68 Fix Sidekiq warning when pushing DMs to direct timeline 2022-01-28 09:07:56 +01:00
Claire b2915613fb Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Upstream-updated lib textually too close to glitch-soc-only dep.
  Updated like upstream.
2022-01-28 08:58:32 +01:00
Claire 03d59340da
Fix Sidekiq warnings about JSON serialization (#17381)
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
Claire ad6ddb9bdd Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `config/environments/production.rb`:
  Upstream changed a header but we had different default headers.
  Applied the same change, and also dropped HSTS headers redundant with
  Rails'.
2022-01-26 22:32:21 +01:00
Claire 166cc5b89d
Fix local distribution of edited statuses (#17380)
Because `FanOutOnWriteService#update?` was broken, edits were considered as new
toots and a regular `update` payload was sent.
2022-01-26 20:53:50 +01:00
Eugen Rochko 6505b39e5d
Fix poll updates being saved as status edits (#17373)
Fix #17344
2022-01-26 18:05:39 +01:00
Claire fe89554a54 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/lib/activitypub/activity/create.rb`:
  Upstream refactored how `Create` activities are handled and how values are
  extracted from `Create`d objects. This conflicted with how glitch-soc
  supported the `directMessage` flag to explicitly distinguish between
  limited and direct messages.
  Ported glitch-soc's changes to latest upstream changes.
- `app/services/fan_out_on_write_service.rb`:
  Upstream largely refactored that file and changed some of the logic.
  This conflicted with glitch-soc's handling of the direct timeline and
  the options to allow replies and boosts in public feeds.
  Ported those glitch-soc changes on top of latest upstream changes.
- `app/services/process_mentions_service.rb`:
  Upstream refactored to move mention-related ActivityPub deliveries to
  `ActivityPub::DeliveryWorker`, while glitch-soc contained an extra check
  to not send local-only toots to remote mentioned users.
  Took upstream's version, as the check is not needed anymore, since it is
  performed at the `ActivityPub::DeliveryWorker` call site already.
- `app/workers/feed_insert_worker.rb`:
  Upstream added support for `update` toot events, while glitch-soc had
  support for an extra timeline support, `direct`.
  Ported upstream changes and extended them to the `direct` timeline.

Additional changes:
- `app/lib/activitypub/parser/status_parser.rb`:
  Added code to handle the `directMessage` flag and take it into account
  to compute visibility.
- `app/lib/feed_manager.rb`:
  Extended upstream's support of `update` toot events to glitch-soc's
  `direct` timeline.
2022-01-19 23:52:48 +01:00
Eugen Rochko d412a8d1f2
Fix error when processing poll updates (#17333)
Regression from #16697
2022-01-19 22:50:01 +01:00
Eugen Rochko 1060666c58
Add support for editing for published statuses (#16697)
* Add support for editing for published statuses

* Fix references to stripped-out code

* Various fixes and improvements

* Further fixes and improvements

* Fix updates being potentially sent to unauthorized recipients

* Various fixes and improvements

* Fix wrong words in test

* Fix notifying accounts that were tagged but were not in the audience

* Fix mistake
2022-01-19 22:37:27 +01:00
Claire c42938aed4 Merge branch 'main' into glitch-soc/merge-upstream 2022-01-19 12:32:44 +01:00
Jeong Arm 2d1f082bb6
Fix NameError on ActivityPub::FetchFeaturedCollectionService (#17326)
Related: #16954
2022-01-19 04:08:46 +01:00
Claire b3bf32a21e Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/views/admin/reports/show.html.haml`:
  Conflicts due to glitch-soc's theming system.
2022-01-17 10:45:25 +01:00
Eugen Rochko 14f436c457
Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
Claire 2d4b4b0b45 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/accounts_controller.rb`:
  Upstream introduced support for private pinned toots, but glitch-soc's query
  was a bit different as it filtered out local-only toots.
  Used upstream's query, while adding local-only filtering back.
- `app/controllers/activitypub/collections_controller.rb`:
  Same thing with regards to local-only posts.
- `app/validators/status_pin_validator.rb`:
  Not a real conflict, but the line below was different in glitch-soc due to
  the configurable pinned toots limit.
2022-01-17 08:28:52 +01:00
Claire d5c9feb7b7
Add support for private pinned posts (#16954)
* Add support for private pinned toots

* Allow local user to pin private toots

* Change wording to avoid "direct message"
2022-01-17 00:49:55 +01:00
Claire fca50e4b50 Merge branch 'main' into glitch-soc/merge-upstream 2022-01-07 19:51:47 +01:00
Jeong Arm d3db2eb7fb
Remove custom emojis on domain purge (#17210) 2021-12-30 08:41:09 +01:00
Claire d4f315f304 Merge branch 'main' into glitch-soc/merge-upstream 2021-12-18 11:44:54 +01:00
Claire 7f803c41e2
Add ability to purge undeliverable domains from admin interface (#16686)
* Add ability to purge undeliverable domains from admin interface

* Add tests
2021-12-17 23:01:21 +01:00
Claire 9b861d56a9 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `.env.production.sample`:
  Copied upstream changes.
- `app/controllers/settings/identity_proofs_controller.rb`:
  Minor conflict due to glitch-soc's extra “enable_keybase” setting.
  Upstream removed keybase support altogether, so did the same.
- `app/controllers/well_known/keybase_proof_config_controller.rb`:
  Minor conflict due to glitch-soc's extra “enable_keybase” setting.
  Upstream removed keybase support altogether, so did the same.
- `lib/mastodon/statuses_cli.rb`:
  Minor conflict due to an optimization that wasn't shared between
  the two versions. Copied upstream's version.
2021-11-26 22:53:55 +01:00
Eugen Rochko 7de0ee7aba
Remove Keybase integration (#17045) 2021-11-26 05:58:18 +01:00
Claire 443ec4f8ba Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/views/admin/tags/index.html.haml`:
  Removed upstream while it had changes in glitch-soc to accomodate for the
  theming system.

Additional changes to accomodate for the theming system:
- `app/views/admin/trends/links/preview_card_providers/index.html.haml`
- `app/views/admin/trends/links/index.html.haml`
- `app/views/admin/trends/tags/index.html.haml`
- `app/views/admin/tags/show.html.haml`
2021-11-25 23:50:35 +01:00
Claire 013bee6afb
Fix filtering DMs from non-followed users (#17042) 2021-11-25 23:46:30 +01:00
Eugen Rochko 6e50134a42
Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire 2b4c7beb73 Merge branch 'main' into glitch-soc/merge-upstream 2021-11-19 18:51:41 +01:00
Takeshi Umeda 3419d3ec84
Bump chewy from 5.2.0 to 7.2.3 (supports Elasticsearch 7.x) (#16915)
* Bump chewy from 5.2.0 to 7.2.2

* fix style (codeclimate)

* fix style

* fix style

* Bump chewy from 7.2.2 to 7.2.3
2021-11-18 22:02:08 +01:00
Claire 4bb6b1a1e7 Merge branch 'main' into glitch-soc/merge-upstream 2021-11-06 00:15:38 +01:00
Eugen Rochko 39cdf61ab7
Add support for structured data and more OpenGraph tags to link cards (#16938)
Save preview cards under their canonical URL

Increase max redirects to follow from 2 to 3
2021-11-05 23:23:05 +01:00
Claire e855df149b Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `.env.production.sample`:
  Upstream added new configuration options, uncommented by default.
  Commented them.
- `Gemfile.lock`:
  Upstream updated dependencies textually close to glitch-soc-specific
  dependencies.
  Updated those upstream dependencies.
2021-10-25 20:28:51 +02:00
Claire ec059317fa
Fix some link previews being incorrectly generated from other prior links (#16885)
* Add tests

* Fix some link previews being incorrectly generated from different prior links

PR #12403 added a cache to avoid redundant queries when the OEmbed endpoint can
be guessed from the URL. This caching mechanism is not perfectly correct as
there is no guarantee that all pages from a given domain share the same
OEmbed provider endpoint.

This PR prevents the FetchOEmbedService from caching OEmbed endpoint that
cannot be generalized by replacing a fully-qualified URL from the endpoint's
parameters, greatly reducing the number of incorrect cached generalizations.
2021-10-21 20:39:35 +02:00
Claire 3f9b28ce26
Add support for fetching Create and Announce activities by URI (#16383)
* Add support for fetching Create and Announce activities by URI

This should improve compatibility with ZAP and offer a way to fetch boosts,
which is currently not possible.

* Add tests
2021-10-21 01:14:04 +02:00
Takeshi Umeda 17f4e457b3
Add remove from followers api (#16864)
* Add followed_by? to account_interactions

* Add RemoveFromFollowersService

* Fix AccountBatch to use RemoveFromFollowersService

* Add remove from followers API
2021-10-18 12:02:35 +02:00
Claire 694c073d1f Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/admin/dashboard_controller.rb`:
  Upstream completely redesigned the admin dashboard.
  glitch-soc tracked extra features, but that list is
  gone.
  Followed upstram.
- `app/views/admin/dashboard/index.html.haml`
  Upstream completely redesigned the admin dashboard.
  glitch-soc tracked extra features, but that list is
  gone.
  Followed upstram.
2021-10-14 21:44:57 +02:00
Claire 216570ad98
Fix scheduled statuses decreasing statuses counts (#16791)
* Add tests

* Fix scheduled statuses decreasing statuses counts

Fixes #16774
2021-10-14 19:59:21 +02:00
Claire 3622110778 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `streaming/index.js`:
  Filtering code for streaming notifications has been refactored upstream, but
  glitch-soc had similar code for local-only toots in the same places.
  Ported upstream changes, but did not refactor local-only filtering.
2021-09-26 18:28:59 +02:00
Eugen Rochko a0d4129893
Refactor notifications to go through a separate stream in streaming API (#16765)
Eliminate need to have custom notifications filtering logic in the
streaming API code by publishing notifications into a separate stream
and then simply using the multi-stream capability to subscribe to that
stream when necessary
2021-09-26 13:23:28 +02:00
Claire 4e2a8c9b38 Merge branch 'main' into glitch-soc/merge-upstream 2021-08-20 13:01:50 +02:00
Claire 9ac7e6fef7
Fix remotely-suspended accounts' toots being merged back into timelines (#16628)
* Fix remotely-suspended accounts' toots being merged back into timelines

* Mark remotely-deleted accounts as remotely suspended
2021-08-20 07:40:33 +01:00
Holger 0cae6c07bb
Fix #16603 (#16605)
Fix issue #16603 undefined method `serialize_payload' for Unsuspend Account Service error.
It seems that this service forgot to `include Payloadable` so that `serialize_payload` could not be found in this service.
2021-08-20 07:39:37 +01:00
Claire 8ec4be4233 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, upstream-updated dependency (redis) textually too
  close to glitch-soc-only dependecy.
  Updated redis gem like upstream did.
2021-08-09 23:25:49 +02:00
Claire 4ac78e2a06
Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Claire 763ab0c7eb
Fix owned account notes not being deleted when an account is deleted (#16579)
* Add account_notes relationship

* Add tests

* Fix owned account notes not being deleted when an account is deleted

* Add post-migration to clean up orphaned account notes
2021-08-08 15:29:57 +02:00
Claire b5e10d07ff Merge branch 'main' into glitch-soc/merge-upstream 2021-07-25 12:48:48 +02:00
Claire 1d67acb72f
Fix scoped order warning in RemoveStatusService (#16531)
Fixes “Scoped order is ignored, it's forced to be batch order.”
2021-07-24 14:41:46 +02:00
Claire d8629e7b86
Add logging of S3-related errors (#16381) 2021-07-21 18:34:39 +02:00
Claire 5f334807d2 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-22 13:11:31 +02:00
Claire 9a19227f17
Fix some RedisLocks auto-releasing too fast (#16276)
* Fix Delete and Create-related locks expiring too fast

Fixes #16238

By default, RedisLock expires after 10 seconds, which may not be enough to
process statuses, especially when those have attached media files.

This commit extends those 10 seconds to 15 minutes, which should be plenty
enough to handle any status, while being short enough to not waste many
sidekiq job retries in the exceedingly rare case in which a sidekiq process
would crash when processing a `Create` or `Delete`.

* Fix other RedisLock autorelease durations

Fixes #15645

- things that only perform a few simple database queries (e.g. finding and
  saving a record) have been left unchanged, so they'll still use the default
  10s duration
- things that perform significantly more complex database queries have been
  changed to a 5 minutes timeout
- things that perform multiple HTTP queries have been changed to a 15 minutes
  timeout
2021-05-19 23:52:08 +02:00
Claire dc58d02192 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-12 11:03:05 +02:00
Takeshi Umeda c403c3695b
Fix to be able to redownload avatar and header (#16190)
* Fix to reset if header and avatar download fails

* Add RedownloadAvatarWorker and RedownloadHeaderWorker
2021-05-11 14:19:22 +02:00
Claire 7ac8bcf209 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-10 21:45:32 +02:00
Claire afb7882189
Fix blocking someone not clearing up list feeds (#16205) 2021-05-10 17:31:55 +02:00
Claire 87c3a0d0b3 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-09 10:41:28 +02:00
Takeshi Umeda 9da5e0b350
Fix webfinger_update_due to run WebFinger on stale activitypub-account (#16182) 2021-05-08 16:22:18 +02:00
Claire 50b430d9a2 Merge branch 'main' into glitch-soc/merge-upstream
- `app/views/statuses/_simple_status.html.haml`:
  Small markup change in glitch-soc, on a line that has been modified by
  upstream. Ported upstream changes.
2021-05-07 18:21:59 +02:00
Eugen Rochko 74081433d0
Change trending hashtags to be affected be reblogs (#16164)
If a status with a hashtag becomes very popular, it stands to
reason that the hashtag should have a chance at trending

Fix no stats being recorded for hashtags that are not allowed
to trend, and stop ignoring bots

Remove references to hashtags in profile directory from the code
and the admin UI
2021-05-07 14:33:43 +02:00
Eugen Rochko 2c77d97e0d
Add joined date to profiles in web UI (#16169) 2021-05-07 14:33:19 +02:00
Eugen Rochko fab65848d2
Fix empty home feed before first follow has finished processing (#16152)
Change queue of merge worker from pull to default
2021-05-04 04:45:08 +02:00
Eugen Rochko f627d2eb93
Fix trying to fetch key from empty URI when verifying HTTP signature (#16100) 2021-05-01 23:18:59 +02:00
Eugen Rochko daccc07dc1
Change auto-following admin-selected accounts, show in recommendations (#16078) 2021-04-24 17:01:43 +02:00
Claire 536892b8ae Merge branch 'main' into glitch-soc/merge-upstream 2021-04-21 13:52:03 +02:00
Claire 0b36e3419d
Fix processing of remote Delete activities (#16084)
* Add tests

* Ensure deleted statuses are marked as such

* Save some redis memory by not storing URIs in delete_upon_arrival values

* Avoid possible race condition when processing incoming Deletes

* Avoid potential duplicate Delete forwards

* Lower lock durations to reduce issues in case of hard crash of the Rails process

* Check for `lock.aquired?` and improve comment

* Refactor RedisLock usage in app/lib/activitypub

* Fix using incorrect or non-existent sender for relaying Deletes
2021-04-21 04:46:09 +02:00
Claire e2a2bc9021 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Upstream updated copyright year, we don't mention it so kept our version.
- `app/controllers/admin/dashboard_controller.rb`:
  Not really a conflict, upstream change (removing the spam checker) too close
  to glitch-soc changes. Ported upstream changes.
- `app/models/form/admin_settings.rb`:
  Same.
- `app/services/remove_status_service.rb`:
  Same.
- `app/views/admin/settings/edit.html.haml`:
  Same.
- `config/settings.yml`:
  Same.
- `config/environments/production.rb`:
  Not a real conflict, upstream added a default HTTP header, but we have
  extra headers in glitch-soc.
  Added the header.
2021-04-20 12:17:14 +02:00
Eugen Rochko ca3bc1b09f
Refactor StatusReachFinder to handle followers and relays as well (#16051) 2021-04-17 15:41:57 +02:00
Eugen Rochko 6d6000f61f
Fix remote reporters not receiving suspend/unsuspend activities (#16050) 2021-04-17 14:55:46 +02:00
Eugen Rochko dde8739020
Fix reports of already suspended accounts being recorded (#16047) 2021-04-16 22:01:05 +02:00
Eugen Rochko 619fad6cf8
Remove spam check and dependency on nilsimsa gem (#16011) 2021-04-11 11:22:50 +02:00
Claire f60c99a8fb Merge branch 'main' into glitch-soc/merge-upstream 2021-03-24 10:53:50 +01:00
Claire cbd0ee1d07
Update Mastodon to Rails 6.1 (#15910)
* Update devise-two-factor to unreleased fork for Rails 6 support

Update tests to match new `rotp` version.

* Update nsa gem to unreleased fork for Rails 6 support

* Update rails to 6.1.3 and rails-i18n to 6.0

* Update to unreleased fork of pluck_each for Ruby 6 support

* Run "rails app:update"

* Add missing ActiveStorage config file

* Use config.ssl_options instead of removed ApplicationController#force_ssl

Disabled force_ssl-related tests as they do not seem to be easily testable
anymore.

* Fix nonce directives by removing Rails 5 specific monkey-patching

* Fix fixture_file_upload deprecation warning

* Fix yield-based test failing with Rails 6

* Use Rails 6's index_with when possible

* Use ActiveRecord::Cache::Store#delete_multi from Rails 6

This will yield better performances when deleting an account

* Disable Rails 6.1's automatic preload link headers

Since Rails 6.1, ActionView adds preload links for javascript files
in the Links header per default.

In our case, that will bloat headers too much and potentially cause
issues with reverse proxies. Furhermore, we don't need those links,
as we already output them as HTML link tags.

* Switch to Rails 6.0 default config

* Switch to Rails 6.1 default config

* Do not include autoload paths in the load path
2021-03-24 10:44:31 +01:00
Claire 1b6a21e6bc Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/validators/status_length_validator.rb`:
  Conflict due to glitch-soc's configurable maximum toot chars.
  Ported upstream changes.
2021-03-11 16:08:15 +01:00
Claire 5614e6724e
Fix URL scanning in note length validator and preview card fetching (#15827)
* Add tests

* Fix URL scanning in note length validator and preview card fetching
2021-03-04 00:12:26 +01:00
Claire d8fdbb054e Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/validators/status_length_validator.rb`:
  Upstream changes too close to glitch-soc MAX_CHARS changes, but not a real
  conflict.
  Applied upstream changes.
- `package.json`:
  glitch-soc-only dependency textually too close to a dependency updated
  upstream, not a real conflict.
  Applied upstream changes.
2021-03-02 12:06:58 +01:00
Claire 65db262550
Update twitter-text from 1.14 to 3.1.0 and fix toot character counting (#15382)
* Update twitter-text from 1.14 to 3.1.0

* Disable emoji parsing

* Properly depend on twitter-text for url detection

* Fix some URLs being wrongly detected client-side

* Add test for server-side validation of non-autolinkable URLs

* Fix server-side status length counting
2021-03-02 12:02:56 +01:00
Claire 5f4c0b79c2
Change ResolveAccountService's handling of skip_webfinger (#15750)
* Change ResolveAccountService's handling of skip_webfinger

Change it so it never makes any webfinger query, as the name would imply.

* Add tests

* Change FollowService to not take an URI for target_account

* Restore domain-block check in FollowService

* Fix tests
2021-02-24 06:32:13 +01:00
Claire 5e11f3a6e1 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/styles/mastodon/modal.scss`:
  For some reason we changed the file loading path in glitch-soc,
  but now upstream has completely changed how the logo is loaded.
  Applied upstream changes.
2021-02-12 10:28:32 +01:00
Eugen Rochko f8972d4503
Fix YouTube embeds failing due to YouTube serving wrong OEmbed URLs (#15716) 2021-02-12 05:45:38 +01:00
Claire b81710c02c Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `CONTRIBUTING.md`:
  Not a real conflict, glitch-soc quotes the upstream file, which has been
  changed. Update the quote.
2021-01-26 14:01:30 +01:00
Takeshi Umeda 7f1c56954b
Fix first return value of FetchLinkCardService.html method (#15630) 2021-01-25 09:22:41 +01:00
luigi 7ea9588520
Use Enumerable#filter_map in more places (#15527) 2021-01-22 16:28:15 +01:00
luigi eb51e43fb4
Optimize some regex matching (#15528)
* Use Regex#match?

* Replace =~ too

* Avoid to call match? from Nil

* Keep value of Regexp.last_match
2021-01-22 10:09:08 +01:00
Claire 33d30632fb Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/models/public_feed.rb`:
  Upstream refactored a bit, glitch-soc had specific code for local-only
  statuses.
  Updated glitch-soc's specific code accordingly.
2021-01-11 11:55:42 +01:00
luigi 087ed84367
Optimize map { ... }.compact calls (#15513)
* Optimize map { ... }.compact

using Enumerable#filter_map, supported since Ruby 2.7

* Add poyfill for Enumerable#filter_map
2021-01-10 00:32:01 +01:00
abcang efffdd3778
Fix rubocop config and warnings (#15503)
* disable NewCops

* update TargetRubyVersion

* Fix Lint/MissingSuper for ActiveModelSerializers::Model

* Fix Lint/MissingSuper for feed

* Fix Lint/FloatComparison

* Do not use instance variables
2021-01-07 09:40:55 +01:00
Claire 90528f43bc Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `config/webpack/configuration.js`:
  Upstream updated the `js-yaml` dependency, which changed how to call it.
  Those changes conflicted because that code is pretty different in glitch-soc
  which has to deal with its more complex theming system.
  Proceeded to the same compatibility changes in glitch-soc's code.
- `package.json` and `yarn.lock`:
  Not really a conflict, just glitch-soc-specific dependencies textually too
  close to some dependencies updated upstream.
2021-01-06 18:08:06 +01:00
ThibG f1f96ebf02
Fix being able to import more than allowed number of follows (#15384)
* Fix being able to import more than allowed number of follows

Without this commit, if someone tries importing a second list of accounts to
follow before the first one has been processed, this will queue imports for
the two whole lists, even if they exceed the account's allowed number of
outgoing follows.

This commit changes it so the individual queued imports aren't exempt from
the follow limit check (they remain exempt from the rate-limiting check
though).

* Catch validation errors to not re-queue failed follows

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-26 23:52:46 +01:00
Eugen Rochko e89648574f
Fix error when changing ACL on missing objects during suspension (#15420) 2020-12-24 15:53:45 +01:00
Eugen Rochko ba0b79fc5c
Fix undefined method error when batch-removing statuses with ElasticSearch on (#15421) 2020-12-24 15:53:26 +01:00
ThibG de57efd055
Fix mentions not being deleted efficiently (#15416)
As a regression from the recent optimizations, mentions were left untouched
until `account.destroy`, which would then delete them individually,
and executing queries to find and delete associated notifications, resulting
in a massive slowdown.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-23 16:43:11 +01:00
Eugen Rochko 473abc35a1
Fix trying to privatize empty media attachments (#15414) 2020-12-23 15:47:03 +09:00
Claire 537afa00f3 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/lib/feed_manager.rb`:
  Not a real conflict, glitch-soc-only DM-related method
  too close to changed upstream stuff.
  Ported upstream changes.
- `app/services/batched_remove_status_service.rb`:
  Additional logic in glitch-soc to clear DMs from timelines.
  Ported upstream changes and fixed the DM TL clearing logic.
- `app/workers/scheduler/feed_cleanup_scheduler.rb`:
  Additional code in glitch-soc to clear DM timelines.
  Ported upstream changes.
2020-12-23 01:47:45 +01:00
ThibG 814b7775fb
Improve performances of deleting favourites when deleting accounts (#15412)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-23 01:35:02 +01:00
ThibG 2ab3e91eaf
Fix BatchedRemoveStatusService not working without ES in rails console (#15408)
Not a huge deal but may cause surprising failures in custom scripts and
development.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-23 00:04:52 +01:00
ThibG 3249d35bdc
Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
ThibG f18349640b
Fix batch order warnings in BatchedRemoveStatusService (#15409)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 20:24:55 +01:00
Eugen Rochko 9915d11c0d
Fix unnecessary queries when batch-removing statuses, 100x faster (#15387) 2020-12-22 17:13:55 +01:00
Claire 38bfaf8855 Merge branch 'master' into glitch-soc/merge-upstream 2020-12-21 18:28:23 +01:00
ThibG 43961035a9
Fix some notifications not being deleted on poll/status deletion (#15402)
* Fix deleting polls not deleting notifications

* Fix fav notification deletion when deleting a toot

* Refactor DeleteAccountService spec

* Add DeleteAccountService tests for other associations and notifications

* Add favourite handling spec in status removal

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-21 18:22:17 +01:00
ThibG 7bf3c6e57b
Fix AccountDeletionWorker crashing and clogging sidekiq queues (#15380)
* Fix account deletion workers being queued multiple times for a single account

* Fix poll votes being unnecessarily instantiated on poll deletion

* Fix favourites being unnecessarily instantiated on status deletion

* Remove inaccurate comments

* Delete polls instead of destroying them

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-20 18:25:00 +01:00
Claire f9d000ebae Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/services/resolve_url_service.rb`:
  The private toot search by URL hack has been revamped upstream.
  Took upstream's version.
2020-12-19 00:55:12 +01:00
ThibG a60d9335d8
Fix resolving accounts sometimes creating duplicate records for a given AP id (#15364)
* Fix ResolveAccountService accepting mismatching acct: URI

* Set attributes that should be updated regardless of suspension

* Fix key fetching

* Automatically merge remote accounts with duplicate `uri`

* Add tests

* Add "tootctl accounts fix-duplicates"

Finds duplicate accounts sharing a same ActivityPub `id`, re-fetch them and
merge them under the canonical `acct:` URI.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-18 23:26:26 +01:00
Eugen Rochko eb35be0431
Fix follow limit preventing re-following of a moved account (#14207) 2020-12-18 09:18:31 +01:00
ThibG b1feb47055
Improve searching for private toots from URL (#14856)
* Improve searching for private toots from URL

Most of the time, when sharing toots, people use the toot URL rather than
the toot URI, which makes sense since it is the user-facing URL.

In Mastodon's case, the URL and URI are different, and Mastodon does not
have an index on URL, which means searching a private toot by URL is done
with a slow query that will only succeed for very recent toots.

This change gets rid of the slow query, and attempts to guess the URI from
URL instead, as Mastodon's are predictable.

* Add tests

* Only return status with guessed uri if url matches

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-17 06:51:49 +01:00
Claire e4f8679eae Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/models/form/admin_settings.rb`:
  New setting added upstream. Ported it.
- `app/views/statuses/_simple_status.html.haml`:
  Upstream removed RTL classes. Did the same.
- `config/settings.yml`:
  New setting added upstream. Ported it.
2020-12-15 14:27:06 +01:00
ThibG 1390cc194b
Add indication to admin UI of whether a report has been forwarded (#13237)
* Add indication to admin UI of whether a report has been forwarded

* Rework how forwarded status is displayed

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-15 04:30:15 +01:00
Claire 1978f7265e Fix processing of Delete activities on direct messages 2020-12-14 23:05:32 +01:00
Claire f8d867bac4 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/services/remove_status_service.rb`:
  Conflict caused by us having a distinc Direct timeline.
  Ported upstream changes.
- `app/javascript/mastodon/features/compose/components/compose_form.js`:
  Conflict between glitch-soc's variable character limit and upstream
  refactoring that part of the code.
  Ported upstream changes.
2020-12-05 17:33:37 +01:00
Eugen Rochko e7e099d1a0
Fix deletes not reaching every server that interacted with status (#15200)
Extract logic for determining ActivityPub inboxes to send deletes
to to its own class and explicitly include the person the status
replied to (even if not mentioned), people who favourited it, and
people who replied to it (though that one is still not recursive)
2020-11-27 15:48:31 +01:00
Thibaut Girka e71b16ae2b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  glitch-soc-only dependency textually too close to updated upstream
  dependencies. Updated to upsteam dependencies.
2020-11-25 15:31:45 +01:00
ThibG 66832cbf33
Undo outgoing follows when suspending a remote account (#15188) 2020-11-21 21:27:00 +01:00
Thibaut Girka 48f0f3ffee Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `.github/ISSUE_TEMPLATE/bug_report.md`:
  Upstream added the `bug` label to bug reports.
  Did the same.
- `app/services/fan_out_on_write_service.rb`:
  Upstream put DMs back into timelines, glitch-soc was already doing it.
  Ignored upstream changes.
2020-11-20 13:27:48 +01:00
ThibG 8b8004a962
Fix webfinger redirect handling in ResolveAccountService (#15187)
* Fix webfinger redirect handling in ResolveAccountService

ResolveAccountService#process_webfinger! handled a one-step webfinger
redirection, but only accepting the result if it matched the exact URI passed
as input, defeating the point of a redirection check.

Instead, use the same logic as in `ActivityPub::FetchRemoteAccountService`,
updating the resulting `acct:` URI with the result of the first webfinger
query.

* Add tests
2020-11-19 19:52:06 +01:00
ThibG 96c1e71329
Add import/export feature for bookmarks (#14956)
* Add ability to export bookmarks

* Add support for importing bookmarks

* Add bookmark import tests

* Add bookmarks export test
2020-11-19 17:48:13 +01:00
ThibG 2f6831f318
Fix sending spurious Rejects when processing remote account deletion (#15104)
* Fix sending spurious Rejects when processing remote account deletion

* Make skip_side_effects imply skip_activitypub
2020-11-19 17:39:47 +01:00
Eugen Rochko df1653174b
Add cache buster feature for media files (#15155)
Nginx can be configured to bypass proxy cache when a special header
is in the request. If the response is cacheable, it will replace
the cache for that request. Proxy caching of media files is
desirable when using object storage as a way of minimizing bandwidth
costs, but has the drawback of leaving deleted media files for
a configured amount of cache time. A cache buster can make those
media files immediately unavailable. This especially makes sense
when suspending and unsuspending an account.
2020-11-19 17:38:06 +01:00
ThibG 966593393e
Fix DMs not appearing into timelines (#15182)
Fixes #15179
2020-11-19 00:23:46 +01:00
Thibaut Girka 10bd6f415d Improve searching for private toots from URL
Most of the time, when sharing toots, people use the toot URL rather than
the toot URI, which makes sense since it is the user-facing URL.

In Mastodon's case, the URL and URI are different, and Mastodon does not
have an index on URL, which means searching a private toot by URL is done
with a slow query that will only succeed for very recent toots.

This change gets rid of the slow query, and attempts to guess the URI from
URL instead, as Mastodon's are predictable.
2020-11-12 23:44:30 +01:00
Thibaut Girka ecd83f495c Merge branch 'master' into glitch-soc/merge-upstream 2020-11-09 10:53:53 +01:00
ThibG 1e64666662
Fix crash in SuspendAccountWorker (#15106)
* Fix crash in SuspendAccountWorker

`follows` is an array thanks to `to_a`

* Fix code style issue

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
2020-11-08 18:29:48 +01:00
Thibaut Girka 0437d70628 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/follower_accounts_controller.rb`:
  Conflict due to upstream changing suspension logic while
  glitch-soc has an extra option to hide followers count.
  Ported upstream changes.
2020-11-08 14:20:35 +01:00
Eugen Rochko 3134691948
Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
Thibaut Girka 412218af2e Merge branch 'master' into glitch-soc/merge-upstream 2020-11-07 18:23:02 +01:00
ThibG ee8cf246cf
Fix crashes in SuspendAccountService/UnsuspendAccountService (#15100)
* Fix crashes in SuspendAccountService/UnsuspendAccountService

* Catch filesystem errors
2020-11-07 13:16:54 +01:00
ThibG f84239ecab
Fix suspension/unsuspension not working because of FeedManager change (#15099) 2020-11-07 13:16:00 +01:00
Thibaut Girka 9c34090db8 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/services/process_mentions_service.rb`:
  glitch-soc local-only handling, ported upstream changes
2020-10-26 15:48:16 +01:00
ThibG 1f945e7390
Fix followers synchronization mechanism not being triggered on mentions (#15026)
e.g. if someone on an instance that previously had followers gets mentioned
in a private toot, before this PR, they would not receive a
Collection-Synchronization header and may show the toot to the former followers
in addition to the mentioned person.
2020-10-23 14:22:16 +02:00
ThibG fb586584f2
Fix account processing failing because of large collections (#15027)
Fixes #15025
2020-10-23 14:21:31 +02:00
Thibaut Girka ec49aa8175 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `.github/dependabot.yml`:
  Updated upstream, we deleted it to not be flooded by Depandabot.
  Kept deleted.
- `Gemfile.lock`:
  Puma updated on both sides, went for the most recent version.
- `app/controllers/api/v1/mutes_controller.rb`:
  Upstream updated the serializer to support timed mutes, while
  glitch-soc added a custom API ages ago to get information that
  is already available elsewhere.
  Dropped the glitch-soc-specific API, went with upstream changes.
- `app/javascript/core/admin.js`:
  Conflict due to changing how assets are loaded. Went with upstream.
- `app/javascript/packs/public.js`:
  Conflict due to changing how assets are loaded. Went with upstream.
- `app/models/mute.rb`:
  🤷
- `app/models/user.rb`:
  New user setting added upstream while we have glitch-soc-specific
  user settings. Added upstream's user setting.
- `config/settings.yml`:
  Upstream added a new user setting close to a user setting we had
  changed the defaults for. Added the new upstream setting.
- `package.json`:
  Upstream dependency updated “too close” to a glitch-soc-specific
  dependency. No real conflict. Updated the dependency.
2020-10-21 19:10:50 +02:00
ThibG ca56527140
Add follower synchronization mechanism (#14510)
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
OSAMU SATO 96761752ec
Add duration parameter to muting. (#13831)
* Adding duration to muting.

* Remove useless checks
2020-10-13 01:01:14 +02:00
Eugen Rochko 5e1364c448
Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
Eugen Rochko 7d985f2aac
Remove dependency on goldfinger gem (#14919)
There are edge cases where requests to certain hosts timeout when
using the vanilla HTTP.rb gem, which the goldfinger gem uses. Now
that we no longer need to support OStatus servers, webfinger logic
is so simple that there is no point encapsulating it in a gem, so
we can just use our own Request class. With that, we benefit from
more robust timeout code and IPv4/IPv6 resolution.

Fix #14091
2020-10-08 00:34:57 +02:00
Thibaut Girka a7aedebc31 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, upstream updated dependencies that were too close to
  glitch-soc-only ones in the file.
- `app/controllers/oauth/authorized_applications_controller.rb`:
  Upstream changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/controllers/settings/base_controller.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/controllers/settings/sessions_controller.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/models/user.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc not preventing moved accounts from logging
  in.
  Ported upstream changes while keeping the ability for moved accounts to log
  in.
- `app/policies/status_policy.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's local-only toots.
  Ported upstream changes.
- `app/serializers/rest/account_serializer.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's ability  to hide followers count.
  Ported upstream changes.
- `app/services/process_mentions_service.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's local-only toots.
  Ported upstream changes.
- `package.json`:
  Not a real conflict, upstream updated dependencies that were too close to
  glitch-soc-only ones in the file.
2020-09-28 14:13:30 +02:00
Eugen Rochko 974b1b79ce
Add option to be notified when a followed user posts (#13546)
* Add bell button

Fix #4890

* Remove duplicate type from post-deployment migration

* Fix legacy class type mappings

* Improve query performance with better index

* Fix validation

* Remove redundant index from notifications
2020-09-18 17:26:45 +02:00
Eugen Rochko ed099d8bdc
Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
Thibaut Girka 9748f074a3 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/api/v1/timelines/public_controller.rb
- app/lib/feed_manager.rb
- app/models/status.rb
- app/services/precompute_feed_service.rb
- app/workers/feed_insert_worker.rb
- spec/models/status_spec.rb

All conflicts are due to upstream refactoring feed management and us having
local-only toots on top of that. Rewrote local-only toots management for
upstream's changes.
2020-09-08 16:26:47 +02:00
Eugen Rochko 65760f59df
Refactor feed manager (#14761) 2020-09-08 03:41:16 +02:00
Eugen Rochko e8bc187845
Refactor how public and tag timelines are queried (#14728) 2020-09-07 11:02:04 +02:00
Thibaut Girka 3fefb7909b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `config/webpack/shared.js`:
  Upstream has changed how Tesseract.js gets included and dropped a dependency.
  The conflict is caused by glitch-soc having different code due to its
  theming system.
  Ported upstream changes.
- `lib/mastodon/version.rb`:
  Upstream refactor/code style change in a place we replaced upstream's
  repo URL with ours.
  Ported upstram changes, keeping our repo URL.
- `yarn.lock`:
  Upstream dropped dependencies, one of which was textually too close to
  a glitch-soc-specific dependency. Not a real conflict.
2020-09-01 17:02:18 +02:00