Commit Graph

478 Commits (62f386d44e4915431d44f834c6767ad3a9e6dbc2)

Author SHA1 Message Date
Claire daab277c0e Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Upstream-updated lib textually too close to glitch-soc-only dep.
  Updated like upstream.
2022-01-28 08:58:32 +01:00
Claire 33ea1c9008 Fix Sidekiq warnings about JSON serialization (#17381)
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
Claire 01e495563e Merge branch 'main' into glitch-soc/merge-upstream 2022-01-19 23:52:53 +01:00
Claire 686eaacbc8 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/lib/activitypub/activity/create.rb`:
  Upstream refactored how `Create` activities are handled and how values are
  extracted from `Create`d objects. This conflicted with how glitch-soc
  supported the `directMessage` flag to explicitly distinguish between
  limited and direct messages.
  Ported glitch-soc's changes to latest upstream changes.
- `app/services/fan_out_on_write_service.rb`:
  Upstream largely refactored that file and changed some of the logic.
  This conflicted with glitch-soc's handling of the direct timeline and
  the options to allow replies and boosts in public feeds.
  Ported those glitch-soc changes on top of latest upstream changes.
- `app/services/process_mentions_service.rb`:
  Upstream refactored to move mention-related ActivityPub deliveries to
  `ActivityPub::DeliveryWorker`, while glitch-soc contained an extra check
  to not send local-only toots to remote mentioned users.
  Took upstream's version, as the check is not needed anymore, since it is
  performed at the `ActivityPub::DeliveryWorker` call site already.
- `app/workers/feed_insert_worker.rb`:
  Upstream added support for `update` toot events, while glitch-soc had
  support for an extra timeline support, `direct`.
  Ported upstream changes and extended them to the `direct` timeline.

Additional changes:
- `app/lib/activitypub/parser/status_parser.rb`:
  Added code to handle the `directMessage` flag and take it into account
  to compute visibility.
- `app/lib/feed_manager.rb`:
  Extended upstream's support of `update` toot events to glitch-soc's
  `direct` timeline.
2022-01-19 23:52:48 +01:00
Eugen Rochko a427958026 Fix error when using raw distribution worker (#17334)
Regression from #16697
2022-01-19 23:05:59 +01:00
Eugen Rochko 06b698a723 Add support for editing for published statuses (#16697)
* Add support for editing for published statuses

* Fix references to stripped-out code

* Various fixes and improvements

* Further fixes and improvements

* Fix updates being potentially sent to unauthorized recipients

* Various fixes and improvements

* Fix wrong words in test

* Fix notifying accounts that were tagged but were not in the audience

* Fix mistake
2022-01-19 22:37:27 +01:00
Claire 485cf5c0c1 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/views/admin/reports/show.html.haml`:
  Conflicts due to glitch-soc's theming system.
2022-01-17 10:45:25 +01:00
Eugen Rochko 2fb76550a9 Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
Claire de7f936b96 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/accounts_controller.rb`:
  Upstream introduced support for private pinned toots, but glitch-soc's query
  was a bit different as it filtered out local-only toots.
  Used upstream's query, while adding local-only filtering back.
- `app/controllers/activitypub/collections_controller.rb`:
  Same thing with regards to local-only posts.
- `app/validators/status_pin_validator.rb`:
  Not a real conflict, but the line below was different in glitch-soc due to
  the configurable pinned toots limit.
2022-01-17 08:28:52 +01:00
Claire 5aade2baac Add support for private pinned posts (#16954)
* Add support for private pinned toots

* Allow local user to pin private toots

* Change wording to avoid "direct message"
2022-01-17 00:49:55 +01:00
Claire 577c74d422 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Upstream added some text, but our README is completely different.
  Kept our README unchanged.
2022-01-16 18:11:20 +01:00
Eugen Rochko 7b3972c270 Remove IP tracking columns from users table (#16409) 2022-01-16 13:23:50 +01:00
Claire 65987b3956 Merge branch 'main' into glitch-soc/merge-upstream 2021-12-18 11:44:54 +01:00
Claire 44c8b6ff2f Add ability to purge undeliverable domains from admin interface (#16686)
* Add ability to purge undeliverable domains from admin interface

* Add tests
2021-12-17 23:01:21 +01:00
Claire 179f1d3a78 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/views/admin/pending_accounts/index.html.haml`:
  Removed upstream, while it had glitch-soc-specific changes to accomodate
  for glitch-soc's theming system.
  Removed the file.

Additional changes:
- `app/views/admin/accounts/index.html.haml':
  Accomodate for glitch-soc's theming system.
2021-12-16 16:20:44 +01:00
Takeshi Umeda 794d9d267d Fix follow recommendation biased towards older accounts (#17126) 2021-12-13 23:21:14 +01:00
Claire 784e38f70b Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/views/admin/tags/index.html.haml`:
  Removed upstream while it had changes in glitch-soc to accomodate for the
  theming system.

Additional changes to accomodate for the theming system:
- `app/views/admin/trends/links/preview_card_providers/index.html.haml`
- `app/views/admin/trends/links/index.html.haml`
- `app/views/admin/trends/tags/index.html.haml`
- `app/views/admin/tags/show.html.haml`
2021-11-25 23:50:35 +01:00
Eugen Rochko ad73becf3e Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire 21e3306721 Merge branch 'main' into glitch-soc/merge-upstream 2021-11-06 00:15:38 +01:00
Claire 925adbf7af Fix AccountNote not having a maximum length (#16942) 2021-11-06 00:12:25 +01:00
Claire 9957c6e8cb Merge branch 'main' into glitch-soc/merge-upstream 2021-08-20 13:01:50 +02:00
Claire 1692e0b381 Fix followers synchronization mechanism not working when URI has empty path (#16510)
* Fix followers synchronization mechanism not working when URI has empty path

To my knowledge, there is no current implementation on the fediverse
that can use bare domains (e.g., actor is at https://example.org instead of
something like https://example.org/actor) that also plans to support the
followers synchronization mechanism. However, Mastodon's current implementation
would exclude such accounts from followers list.

Also adds tests and rename them to reflect the proper method names.

* Move url prefix regexp to its own constant
2021-08-11 17:48:42 +02:00
Claire 8391d975d7 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, upstream-updated dependency (redis) textually too
  close to glitch-soc-only dependecy.
  Updated redis gem like upstream did.
2021-08-09 23:25:49 +02:00
Claire 1fcf310c60 Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Takeshi Umeda 496945f4c4 Fix when MoveWorker cannot get locale from remote account (#16576) 2021-08-08 15:31:02 +02:00
Claire f0035140d7 Merge branch 'main' into glitch-soc/merge-upstream 2021-06-24 08:08:02 +02:00
Claire 4da47fc09f Fix newlines in accout notes added by the Move handler (#16415)
* Fix newlines in account notes added by the move handler

* Make MoveWorker more robust
2021-06-23 23:55:47 +02:00
Claire 1e5591bdde Merge branch 'main' into glitch-soc/merge-upstream 2021-06-23 10:46:24 +02:00
Eugen Rochko 2067b0bf34 Add authentication history (#16408) 2021-06-21 17:07:30 +02:00
Claire f81f50023d Merge branch 'main' into glitch-soc/merge-upstream 2021-05-22 13:11:31 +02:00
Claire 45903ae80a Fix some RedisLocks auto-releasing too fast (#16276)
* Fix Delete and Create-related locks expiring too fast

Fixes #16238

By default, RedisLock expires after 10 seconds, which may not be enough to
process statuses, especially when those have attached media files.

This commit extends those 10 seconds to 15 minutes, which should be plenty
enough to handle any status, while being short enough to not waste many
sidekiq job retries in the exceedingly rare case in which a sidekiq process
would crash when processing a `Create` or `Delete`.

* Fix other RedisLock autorelease durations

Fixes #15645

- things that only perform a few simple database queries (e.g. finding and
  saving a record) have been left unchanged, so they'll still use the default
  10s duration
- things that perform significantly more complex database queries have been
  changed to a 5 minutes timeout
- things that perform multiple HTTP queries have been changed to a 15 minutes
  timeout
2021-05-19 23:52:08 +02:00
Claire eb67ebb939 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-12 11:03:05 +02:00
Takeshi Umeda 43eeefdfcf Fix to be able to redownload avatar and header (#16190)
* Fix to reset if header and avatar download fails

* Add RedownloadAvatarWorker and RedownloadHeaderWorker
2021-05-11 14:19:22 +02:00
Claire 31b8010045 Merge branch 'main' into glitch-soc/merge-upstream
- `app/views/statuses/_simple_status.html.haml`:
  Small markup change in glitch-soc, on a line that has been modified by
  upstream. Ported upstream changes.
2021-05-07 18:21:59 +02:00
Claire 86f5fad111 Add Ruby 3.0 support (#16046)
* Fix issues with POSIX::Spawn, Terrapin and Ruby 3.0

Also improve the Terrapin monkey-patch for the stderr/stdout issue.

* Fix keyword argument handling throughout the codebase

* Monkey-patch Paperclip to fix keyword arguments handling in validators

* Change validation_extensions to please CodeClimate

* Bump microformats from 4.2.1 to 4.3.1

* Allow Ruby 3.0

* Add Ruby 3.0 test target to CircleCI

* Add test for admin dashboard warnings

* Fix admin dashboard warnings on Ruby 3.0
2021-05-06 14:22:54 +02:00
Eugen Rochko 115cdee0ed Fix media redownload worker retrying on unexpected response codes (#16111) 2021-05-05 23:46:59 +02:00
Claire 78eddfc83c Improve performance of follow recommendation scheduler (#16159)
Express follow_recommendations in terms of account_summaries rather than
accounts, integrate filters that are unconditionally used, and materialize
the resulting view.

This should result in the bulk of the computation being performed only once
instead of **once per recommendation language**.
2021-05-05 22:04:52 +02:00
Eugen Rochko 876061e777 Fix empty home feed before first follow has finished processing (#16152)
Change queue of merge worker from pull to default
2021-05-04 04:45:08 +02:00
Eugen Rochko bc522c698c Fix thread resolve worker retrying when status no longer exists (#16109) 2021-04-26 18:56:45 +02:00
Eugen Rochko bf5d831fe7 Fix nil error when removing status caused by race condition (#16099) 2021-04-24 13:35:39 +02:00
Claire f5a9971cac Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Upstream updated copyright year, we don't mention it so kept our version.
- `app/controllers/admin/dashboard_controller.rb`:
  Not really a conflict, upstream change (removing the spam checker) too close
  to glitch-soc changes. Ported upstream changes.
- `app/models/form/admin_settings.rb`:
  Same.
- `app/services/remove_status_service.rb`:
  Same.
- `app/views/admin/settings/edit.html.haml`:
  Same.
- `config/settings.yml`:
  Same.
- `config/environments/production.rb`:
  Not a real conflict, upstream added a default HTTP header, but we have
  extra headers in glitch-soc.
  Added the header.
2021-04-20 12:17:14 +02:00
Eugen Rochko a8bf951926 Change Web Push API deliveries to use request pooling (#16014) 2021-04-12 14:25:34 +02:00
Eugen Rochko 92b2d926bf Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
Claire 3c09518a1d Merge branch 'main' into glitch-soc/merge-upstream 2021-03-15 11:26:59 +01:00
Eugen Rochko 9f16bd9c89 Fix configuration for sidekiq-unique-jobs after 7.x upgrade (#15908)
Remove locks from scheduled jobs
2021-03-15 11:17:43 +01:00
Claire 1ddb91d539 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `config/webpack/configuration.js`:
  Upstream updated the `js-yaml` dependency, which changed how to call it.
  Those changes conflicted because that code is pretty different in glitch-soc
  which has to deal with its more complex theming system.
  Proceeded to the same compatibility changes in glitch-soc's code.
- `package.json` and `yarn.lock`:
  Not really a conflict, just glitch-soc-specific dependencies textually too
  close to some dependencies updated upstream.
2021-01-06 18:08:06 +01:00
ThibG 82d014fd08 Fix being able to import more than allowed number of follows (#15384)
* Fix being able to import more than allowed number of follows

Without this commit, if someone tries importing a second list of accounts to
follow before the first one has been processed, this will queue imports for
the two whole lists, even if they exceed the account's allowed number of
outgoing follows.

This commit changes it so the individual queued imports aren't exempt from
the follow limit check (they remain exempt from the rate-limiting check
though).

* Catch validation errors to not re-queue failed follows

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-26 23:52:46 +01:00
Claire c9a63555e1 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/lib/feed_manager.rb`:
  Not a real conflict, glitch-soc-only DM-related method
  too close to changed upstream stuff.
  Ported upstream changes.
- `app/services/batched_remove_status_service.rb`:
  Additional logic in glitch-soc to clear DMs from timelines.
  Ported upstream changes and fixed the DM TL clearing logic.
- `app/workers/scheduler/feed_cleanup_scheduler.rb`:
  Additional code in glitch-soc to clear DM timelines.
  Ported upstream changes.
2020-12-23 01:47:45 +01:00
ThibG 8d75aa630f Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
Claire a9527cca0b Merge branch 'master' into glitch-soc/merge-upstream 2020-12-21 18:28:23 +01:00
ThibG bbf0e7107c Fix AccountDeletionWorker crashing and clogging sidekiq queues (#15380)
* Fix account deletion workers being queued multiple times for a single account

* Fix poll votes being unnecessarily instantiated on poll deletion

* Fix favourites being unnecessarily instantiated on status deletion

* Remove inaccurate comments

* Delete polls instead of destroying them

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-20 18:25:00 +01:00
Claire 48722a3188 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/services/resolve_url_service.rb`:
  The private toot search by URL hack has been revamped upstream.
  Took upstream's version.
2020-12-19 00:55:12 +01:00
ThibG 0b3e9c1e4c Fix resolving accounts sometimes creating duplicate records for a given AP id (#15364)
* Fix ResolveAccountService accepting mismatching acct: URI

* Set attributes that should be updated regardless of suspension

* Fix key fetching

* Automatically merge remote accounts with duplicate `uri`

* Add tests

* Add "tootctl accounts fix-duplicates"

Finds duplicate accounts sharing a same ActivityPub `id`, re-fetch them and
merge them under the canonical `acct:` URI.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-18 23:26:26 +01:00
Claire e590177b29 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/models/form/admin_settings.rb`:
  New setting added upstream. Ported it.
- `app/views/statuses/_simple_status.html.haml`:
  Upstream removed RTL classes. Did the same.
- `config/settings.yml`:
  New setting added upstream. Ported it.
2020-12-15 14:27:06 +01:00
Eugen Rochko df8874b24e Fix performance on instances list in admin UI (#15282)
- Reduce duplicate queries
- Remove n+1 queries
- Add accounts count to detailed view
- Add separate action log entry for updating existing domain blocks
2020-12-14 09:06:34 +01:00
Thibaut Girka 3216868024 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `.github/ISSUE_TEMPLATE/bug_report.md`:
  Upstream added the `bug` label to bug reports.
  Did the same.
- `app/services/fan_out_on_write_service.rb`:
  Upstream put DMs back into timelines, glitch-soc was already doing it.
  Ignored upstream changes.
2020-11-20 13:27:48 +01:00
ThibG b0bc965658 Fix sending spurious Rejects when processing remote account deletion (#15104)
* Fix sending spurious Rejects when processing remote account deletion

* Make skip_side_effects imply skip_activitypub
2020-11-19 17:39:47 +01:00
Eugen Rochko 471b8570ad Add cache buster feature for media files (#15155)
Nginx can be configured to bypass proxy cache when a special header
is in the request. If the response is cacheable, it will replace
the cache for that request. Proxy caching of media files is
desirable when using object storage as a way of minimizing bandwidth
costs, but has the drawback of leaving deleted media files for
a configured amount of cache time. A cache buster can make those
media files immediately unavailable. This especially makes sense
when suspending and unsuspending an account.
2020-11-19 17:38:06 +01:00
Thibaut Girka 90f5d780cb Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/concerns/sign_in_token_authentication_concern.rb`:
  Conflict caused because of glitch-soc's theming system.
  Took upstream's new code and applied the theming system changes on top
  of it.
- `app/controllers/concerns/two_factor_authentication_concern.rb`:
  Conflict caused because of glitch-soc's theming system.
  Took upstream's new code and applied the theming system changes on top
  of it.
2020-11-14 00:30:36 +01:00
ThibG 53ad960562 Fix AccountDeletionWorker not accepting keyword arguments (#15152) 2020-11-13 22:17:04 +01:00
Thibaut Girka 64ef99a31e Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/follower_accounts_controller.rb`:
  Conflict due to upstream changing suspension logic while
  glitch-soc has an extra option to hide followers count.
  Ported upstream changes.
2020-11-08 14:20:35 +01:00
Eugen Rochko 38be9af436 Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
Thibaut Girka ea4b2caf3a Merge branch 'master' into glitch-soc/merge-upstream 2020-11-03 17:28:54 +01:00
ThibG 2b0491bb3c Fix poll ending notifications being created for each vote (#15071)
On a poll ending, notifications were created for each vote instead
of for each voter.
2020-11-01 06:34:43 +01:00
Thibaut Girka 6fdc4e8d47 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `.github/dependabot.yml`:
  Updated upstream, we deleted it to not be flooded by Depandabot.
  Kept deleted.
- `Gemfile.lock`:
  Puma updated on both sides, went for the most recent version.
- `app/controllers/api/v1/mutes_controller.rb`:
  Upstream updated the serializer to support timed mutes, while
  glitch-soc added a custom API ages ago to get information that
  is already available elsewhere.
  Dropped the glitch-soc-specific API, went with upstream changes.
- `app/javascript/core/admin.js`:
  Conflict due to changing how assets are loaded. Went with upstream.
- `app/javascript/packs/public.js`:
  Conflict due to changing how assets are loaded. Went with upstream.
- `app/models/mute.rb`:
  🤷
- `app/models/user.rb`:
  New user setting added upstream while we have glitch-soc-specific
  user settings. Added upstream's user setting.
- `config/settings.yml`:
  Upstream added a new user setting close to a user setting we had
  changed the defaults for. Added the new upstream setting.
- `package.json`:
  Upstream dependency updated “too close” to a glitch-soc-specific
  dependency. No real conflict. Updated the dependency.
2020-10-21 19:10:50 +02:00
ThibG 694a5e33d5 Add follower synchronization mechanism (#14510)
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
OSAMU SATO ba9252b32d Add duration parameter to muting. (#13831)
* Adding duration to muting.

* Remove useless checks
2020-10-13 01:01:14 +02:00
Eugen Rochko da4c730c47 Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
Thibaut Girka 7a84a84fc1 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, upstream updated dependencies that were too close to
  glitch-soc-only ones in the file.
- `app/controllers/oauth/authorized_applications_controller.rb`:
  Upstream changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/controllers/settings/base_controller.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/controllers/settings/sessions_controller.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/models/user.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc not preventing moved accounts from logging
  in.
  Ported upstream changes while keeping the ability for moved accounts to log
  in.
- `app/policies/status_policy.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's local-only toots.
  Ported upstream changes.
- `app/serializers/rest/account_serializer.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's ability  to hide followers count.
  Ported upstream changes.
- `app/services/process_mentions_service.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's local-only toots.
  Ported upstream changes.
- `package.json`:
  Not a real conflict, upstream updated dependencies that were too close to
  glitch-soc-only ones in the file.
2020-09-28 14:13:30 +02:00
Eugen Rochko 02c4c91faa Add option to be notified when a followed user posts (#13546)
* Add bell button

Fix #4890

* Remove duplicate type from post-deployment migration

* Fix legacy class type mappings

* Improve query performance with better index

* Fix validation

* Remove redundant index from notifications
2020-09-18 17:26:45 +02:00
Eugen Rochko e514304a76 Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
Thibaut Girka 04f2c25e5d Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/api/v1/timelines/public_controller.rb
- app/lib/feed_manager.rb
- app/models/status.rb
- app/services/precompute_feed_service.rb
- app/workers/feed_insert_worker.rb
- spec/models/status_spec.rb

All conflicts are due to upstream refactoring feed management and us having
local-only toots on top of that. Rewrote local-only toots management for
upstream's changes.
2020-09-08 16:26:47 +02:00
Eugen Rochko 38aa7a9a7f Refactor feed manager (#14761) 2020-09-08 03:41:16 +02:00
Thibaut Girka 714a3fcb3b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/mastodon/actions/compose.js`:
  Not a “real” conflict, but change too close to a change we made to
  fix the vanilla WebUI locally pushing authored local-only toots in the
  public TL view.
2020-07-22 13:38:17 +02:00
Takeshi Umeda 63aa1c1fb9 Changed retries and rescued in ActivityPub::ProcessingWorker (#14355)
* Changed the number of retries and rescued exceptions in ActivityPub::ProcessingWorker

* Remove RecordNotUnique from rescue
2020-07-20 11:26:12 +02:00
Thibaut Girka 67657d8361 Merge branch 'master' into glitch-soc/merge-upstream 2020-07-16 14:43:50 +02:00
ThibG 109ea6f860 Fix removing allowed domains being done synchronously (#14302)
* Fix removing allowed domains being done synchronously

* Add tests
2020-07-15 21:08:19 +02:00
Thibaut Girka f5afdaa2e0 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `.env.production.sample`:
  Upstream changed it completely.
  Changed ours to merge upstream's new structure, but
  keeping most of the information.
2020-07-05 19:35:56 +02:00
Eugen Rochko 8517a5fdb4 Add color extraction for audio thumbnails (#14209) 2020-07-05 18:28:25 +02:00
Thibaut Girka 928a3a9fd5 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/packs/public.js`:
  Conflict because part of that file has been split to
  `app/javascript/core/settings.js`. Ported those changes
  there.
2020-07-01 19:23:14 +02:00
Eugen Rochko ca238c879d Change Redis#exists calls to Redis#exists? to avoid deprecation warning (#14191) 2020-07-01 19:05:21 +02:00
ThibG 3e51b04d40 Change move handler to carry blocks over (#14144)
* Change move handler to carry blocks and mutes over

When user A blocks user B and B moves to a new account C, make A block C
accordingly.

Note that it only works if A's instance is aware of the Move, that is,
if B is on A's instance or has followers there.

* Also notify instances with known people blocking you when moving

* Add automatic account notes when blocking/muting an account that had no note
2020-07-01 13:51:15 +02:00
Thibaut Girka 2a2af880b8 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `.env.production.sample`:
  Upstream deleted it, I decided to keep it.
- `package.json`:
  Upstream updated a dependency textually too close to wavesurfer.js
  which has been deleted from upstream but is kept in glitch-soc for now.
2020-06-30 23:00:20 +02:00
ThibG 3afc7977b1 Add user notes on accounts (#14148)
* Add UserNote model

* Add UI for user notes

* Put comment in relationships entity

* Add API to create user notes

* Copy user notes to new account when receiving a Move activity

* Address some of the review remarks

* Replace modal by inline edition

* Please CodeClimate

* Button design changes

* Change design again

* Cancel note edition when pressing Escape

* Fixes

* Tweak design again

* Move “Add note” item, and allow users to add notes to themselves

* Rename UserNote into AccountNote, rename “comment” Relationship attribute to “note”
2020-06-30 19:19:50 +02:00
Eugen Rochko a3ce01a102 Add customizable thumbnails for audio and video attachments (#14145)
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
2020-06-29 13:56:55 +02:00
Thibaut Girka 0e33582bff Merge branch 'master' into glitch-soc/merge-upstream 2020-06-26 13:02:14 +02:00
Eugen Rochko 12ac904a33 Fix various issues around OpenGraph representation of media (#14133)
- Fix audio attachments not being represented in OpenGraph tags
- Fix audio being represented as "1 image" in OpenGraph descriptions
- Fix video metadata being overwritten by paperclip-av-transcoder
- Fix embedded player not using Mastodon's UI
- Fix audio/video progress bars not moving smoothly
- Fix audio/video buffered bars not displaying correctly
2020-06-25 01:33:01 +02:00
Thibaut Girka 83dc54c487 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/activitypub/collections_controller.rb`:
  Conflict due to glitch-soc having to take care of local-only
  pinned toots in that controller.
  Took upstream's changes and restored the local-only special
  handling.
- `app/controllers/auth/sessions_controller.rb`:
  Minor conflicts due to the theming system, applied upstream
  changes, adapted the following two files for glitch-soc's
  theming system:
  - `app/controllers/concerns/sign_in_token_authentication_concern.rb`
  - `app/controllers/concerns/two_factor_authentication_concern.rb`
- `app/services/backup_service.rb`:
  Minor conflict due to glitch-soc having to handle local-only
  toots specially. Applied upstream changes and restored
  the local-only special handling.
- `app/views/admin/custom_emojis/index.html.haml`:
  Minor conflict due to the theming system.
- `package.json`:
  Upstream dependency updated, too close to a glitch-soc-only
  dependency in the file.
- `yarn.lock`:
  Upstream dependency updated, too close to a glitch-soc-only
  dependency in the file.
2020-06-09 10:39:20 +02:00
ThibG a7400a4f97 Make domain block/silence/reject-media code more robust (#13424)
* Split media cleanup from reject-media domain blocks to its own service

* Slightly improve ClearDomainMediaService error handling

* Lower DomainClearMediaWorker to lowest-priority queue

* Do not catch ActiveRecord::RecordNotFound in domain block workers

* Fix DomainBlockWorker spec labels

* Add some specs

* Change domain blocks to immediately mark accounts as suspended

Rather than doing so sequentially, account after account, while cleaning
their data. This doesn't change much about the time the block takes to
complete, but it immediately prevents interaction with the blocked domain,
while up to now, it would only be guaranteed when the process ends.
2020-06-09 10:32:00 +02:00
Takeshi Umeda 8054496ea7 Fix performance of follow import (#13836) 2020-06-09 10:26:58 +02:00
Eugen Rochko b864e78db8 Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
Thibaut Girka 4853a87df8 Merge branch 'master' into glitch-soc/merge-upstream 2020-05-13 22:11:49 +02:00
Takeshi Umeda 15c0903772 Fix RedownloadMediaWorker (#13741) 2020-05-12 15:55:25 +02:00
Thibaut Girka 4e4e5316c1 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, just a glitch-soc-only dependency too close to a
  dependency that got updated upstream. Updated as well.
- `app/models/status.rb`:
  Not a real conflict, just a change too close to glitch-soc-changed code
  for optionally showing boosts in public timelines.
  Applied upstream changes.
- `app/views/layouts/application.html.haml`:
  Upstream a new, static CSS file, conflict due to glitch-soc's theming
  system, include the file regardless of the theme.
- `config/initializers/content_security_policy.rb`:
  Upstream dropped 'unsafe-inline' from the 'style-src' directive, but
  both files are very different. Removed 'unsafe-inline' as well.
2020-05-10 16:19:56 +02:00
Eugen Rochko 909c31a765 Fix error within error when limiting backtrace to 3 lines (#13120)
Fix #13086, close #13113
2020-05-10 10:30:27 +02:00
Thibaut Girka 43a9ed3de8 Merge branch 'master' into glitch-soc/merge-upstream 2020-04-17 20:16:24 +02:00
Takeshi Umeda 8eca96eb77 Fix DeliveryWorker not to call failure_tracker when inbox_url is unavailable (#13482) 2020-04-16 08:04:10 +02:00
Eugen Rochko 2d3219549b Change delivery failure tracking to work with hostnames instead of URLs (#13437) 2020-04-15 20:33:24 +02:00
Thibaut Girka 2f49575839 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  We have a different README than upstream, discarded upstream changes
  and kept ours.
2020-04-12 16:50:27 +02:00
Eugen Rochko 331bc98f25 Fix trying to delete already deleted file when post-processing (#13406)
Fix #13403
2020-04-06 14:11:22 +02:00
Thibaut Girka c13682bc0d Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/packs/public.js`:
  Upstream modified code that we split between multiple files due
  to glitch-soc's theming system.
  Ported those changes.
2020-04-05 18:32:06 +02:00
ThibG ef48ffc127 Fix PostgreSQL load when linking in announcements (#13250)
* Fix PostgreSQL load when linking in announcements

Fixes #13245 by caching status lookups

Since statuses are supposed to be known already and we only
need their URLs and a few other things, caching them should
be fine.

Since it's only used by announcements so far, there won't
be much statuses to cache.

* Perform status lookup when saving announcements, not when rendering them

* Change EntityCache#status to fetch URLs instead of looking into the database

* Move announcement link lookup to publishing worker

* Address issues pointed out during review
2020-04-05 12:51:22 +02:00
Thibaut Girka 631dacf1d7 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/mastodon/features/compose/components/poll_form.js`:
  Upstream bumped poll option character limit, but we already had
  a higher one, kept ours.
- `app/validators/poll_validator.rb`:
  Upstream bumped poll option character limit, but we already had
  a higher one, kept ours.
- `config/initializers/content_security_policy.rb`:
  Upstream added a rule, the way we compute ours is different, but
  that added rule has been ported.
- `package.json`:
  No real conflict, dependency update. Performed the same update.
- `yarn.lock`:
  No real conflict, dependency update. Performed the same update.
2020-04-02 20:32:00 +02:00
Eugen Rochko b1c596e8f3 Fix background jobs not using locks like they are supposed to (#13361)
Also:

- Fix locks not being removed when jobs go to the dead job queue
- Add UI for managing locks to the Sidekiq dashboard
- Remove unused Sidekiq workers

Fix #13349
2020-03-31 21:59:03 +02:00
Thibaut Girka 99fc46d023 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Our README.md files are completely different. Discarded upstream changes.
- `app/javascript/core/admin.js`:
  Updating rails-ujs, no real conflict, but a comment to close to changed
  code. Various glitch-soc-only files have been updated to match those changes,
  though.
- `package.json`:
  No real conflict, just an additional dependency in glitch-soc that was too
  close to something updated upstream. Took upstream's changes.
2020-03-22 16:10:44 +01:00
Eugen Rochko f459919552 Change local media attachments to perform heavy processing asynchronously (#13210)
Fix #9106
2020-03-08 23:56:18 +01:00
Thibaut Girka 9604766a18 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/serializers/rest/account_serializer.rb`:
  Upstream added code too close to glitch-soc-specific followers-hiding code.
  Ported upstream changes.
2020-01-27 15:46:50 +01:00
Eugen Rochko 9fb8165712 Add publish/unpublish controls to announcements in admin UI (#12967) 2020-01-27 11:05:33 +01:00
Eugen Rochko 9d3bba9576 Fix expired announcements being re-published (#12964) 2020-01-26 22:43:18 +01:00
Eugen Rochko dd68e44cfa Add streaming API updates for announcements being modified or deleted (#12963)
Change `all_day` to be a visual client-side cue only

Publish immediately if `scheduled_at` is in the past

Add `published_at` and `updated_at` to announcements JSON
2020-01-26 20:07:26 +01:00
Thibaut Girka f1ceb471b2 Merge branch 'master' into glitch-soc/merge-upstream 2020-01-25 10:04:41 +01:00
ThibG bfc08eca51 Fix type error in announcement reaction update event payload (#12946) 2020-01-24 22:08:24 +01:00
Thibaut Girka a8e43875a5 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/statuses_controller.rb`:
  Minor conflict due to theming system
2020-01-24 14:37:06 +01:00
Eugen Rochko f95fd62bac Add announcements (#12662)
* Add announcements

Fix #11006

* Add reactions to announcements

* Add admin UI for announcements

* Add unit tests

* Fix issues

- Add `with_dismissed` param to announcements API
- Fix end date not being formatted when time range is given
- Fix announcement delete causing reactions to send streaming updates
- Fix announcements container growing too wide and mascot too small
- Fix `all_day` being settable when no time range is given
- Change text "Update" to "Announcement"

* Fix scheduler unpublishing announcements before they are due

* Fix filter params not being passed to announcements filter
2020-01-23 22:00:13 +01:00
Thibaut Girka 82e197787f Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  No real conflict, glitch-soc-only dependency (redcarpet) too close to an
  upstream one (rdf-normalize)
- `README.md`:
  we have different READMEs, discarded upstream's changes
- `app/views/admin/custom_emojis/index.html.haml`:
  No real conflict, different context because of glitch-soc theming
- `lib/mastodon/statuses_cli.rb`:
  Upstream added code to keep bookmarked statuses, we were already doing so
  with slightly different code. Discarded upstream's changes.
- `package.json`:
  No real conflict, glitch-soc-only dependency (favico.js) too close to
  an upstream one
2020-01-12 15:57:34 +01:00
Daigo 3 Dango 18cd3a9cb3 Suppress backtrace when delivering toots (#12798)
This is to suppress irrelevant backtrace from errors raised when
delivering toots to remote servers. The errors are usually out of
control by the local server and backtraces don't provide much
information.

This is similar to https://github.com/tootsuite/mastodon/pull/5174
and shortens backtraces like below:

```
WARN: Mastodon::UnexpectedResponseError: https://example.com/inbox returned code 523
WARN: app/workers/activitypub/delivery_worker.rb:48:in `block (3 levels) in perform_request'
app/lib/request.rb:75:in `perform'
app/workers/activitypub/delivery_worker.rb:47:in `block (2 levels) in perform_request'
app/lib/request_pool.rb:53:in `use'
app/lib/request_pool.rb:108:in `block (2 levels) in with'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/notifications.rb:170:in `instrument'
app/lib/request_pool.rb:107:in `block in with'
app/lib/connection_pool/shared_connection_pool.rb:21:in `block (2 levels) in with'
app/lib/connection_pool/shared_connection_pool.rb:20:in `handle_interrupt'
app/lib/connection_pool/shared_connection_pool.rb:20:in `block in with'
app/lib/connection_pool/shared_connection_pool.rb:16:in `handle_interrupt'
app/lib/connection_pool/shared_connection_pool.rb:16:in `with'
app/lib/request_pool.rb:106:in `with'
app/workers/activitypub/delivery_worker.rb:46:in `block in perform_request'
vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:51:in `run_code'
vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:42:in `run_yellow'
vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:24:in `run'
app/workers/activitypub/delivery_worker.rb:57:in `perform_request'
app/workers/activitypub/delivery_worker.rb:25:in `perform'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:31:in `block in call'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/statsd/publisher.rb:27:in `statsd_time'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:30:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
app/lib/sidekiq_error_handler.rb:5:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/scout_apm-2.3.0.pre3/lib/scout_apm/background_job_integrations/sidekiq.rb:69:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-unique-jobs-6.0.18/lib/sidekiq_unique_jobs/server/middleware.rb:29:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:73:in `block in wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:72:in `wrap'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
```

```
WARN: Stoplight::Error::RedLight: https://example.com/inbox
WARN: vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:46:in `run_red'
vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:25:in `run'
app/workers/activitypub/delivery_worker.rb:57:in `perform_request'
app/workers/activitypub/delivery_worker.rb:25:in `perform'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:31:in `block in call'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/statsd/publisher.rb:27:in `statsd_time'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:30:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
app/lib/sidekiq_error_handler.rb:5:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/scout_apm-2.3.0.pre3/lib/scout_apm/background_job_integrations/sidekiq.rb:69:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-unique-jobs-6.0.18/lib/sidekiq_unique_jobs/server/middleware.rb:29:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:73:in `block in wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:72:in `wrap'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
```
2020-01-11 02:15:03 +01:00
Thibaut Girka 2591cedcad Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/application_controller.rb`:
  Conflict due to theming system.
- `app/controllers/oauth/authorizations_controller.rb`:
  Conflict due to theming system.
2020-01-04 23:04:42 +01:00
ThibG 4fcd1d59a9 Fix RefollowWorker not keeping show_reblogs setting (#12707)
* Fix RefollowWorker not keeping show_reblogs setting

* Fix RefollowWorker
2020-01-02 20:52:39 +01:00
Thibaut Girka 0fc784f8d0 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/mastodon/features/compose/components/poll_form.js`
  conflict because of the poll option limit being different than upstream's
2019-11-07 13:43:56 +01:00
ThibG 28b68ec875 Fix various issues with account migration (#12301)
* Fix being able to follow oneself by moving to an account that was following the old one

* Add specs

* Add spec to catch MoveWorker issue with local followers following both accounts

* Fix move worker breaking when a local account follows both source and target accounts

* Fix migration from remote to local account not sending Undo Follow

* Fix show_reblogs not being preserved for moved account's followers
2019-11-07 09:05:07 +02:00
Thibaut Girka 07abea7ba8 Merge tag 'v3.0.0' into glitch-soc/merge-upstream 2019-10-04 19:49:43 +02:00
Eugen Rochko 1656a9fa9c Fix wrong attribute in MoveWorker (#12066) 2019-10-03 17:50:48 +02:00
Thibaut Girka cc94c90bf1 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- Gemfile
- Gemfile.lock
- app/controllers/about_controller.rb
- app/controllers/auth/sessions_controller.rb
2019-09-30 12:23:57 +02:00
Eugen Rochko ae03161ad9 Fix account migration not affecting followers on origin server (#11980) 2019-09-29 16:23:13 +02:00
Takeshi Umeda 4049193f69 Fixed missing account in MoveDistributionWorker (#11906) 2019-09-21 09:11:58 +02:00
Eugen Rochko a2d3728c36 Add account migration UI (#11846)
Fix #10736

- Change data export to be available for non-functional accounts
- Change non-functional accounts to include redirecting accounts
2019-09-19 20:58:19 +02:00
Thibaut Girka 4837d6275a Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/auth/sessions_controller.rb
  Minor conflict due to glitch-soc's theming code
2019-09-18 17:25:56 +02:00
Eugen Rochko d321515c11 Fix Move handler queuing jobs that will fail if account is suspended (#11864)
Don't put Move handler on cooldown if it didn't run. Skip unmerging
from timelines to save unnecessary work.
2019-09-17 08:44:45 +02:00
abcang f419d968d8 Improve IP cleanup query (#11871) 2019-09-17 08:44:25 +02:00
Eugen Rochko a5e34d5ff8 Fix web push subscriptions being deleted on rate limit or timeout (#11826) 2019-09-13 19:15:47 +02:00
Thibaut Girka 2e38d889a3 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- Gemfile
- app/controllers/api/v1/search_controller.rb
  Conflict because we changed the number of default results to be
  configurable
- app/lib/settings/scoped_settings.rb
  Addition of a new “noindex” site-wide setting,
  conflict due to our change of the two other site-wide settings
  (default flavour and skin instead of theme)
- spec/controllers/application_controller_spec.rb
  Addition of a new “noindex” site-wide setting,
  conflict due to our change of the two other site-wide settings
  (default flavour and skin instead of theme)
2019-09-13 18:13:43 +02:00
Eugen Rochko 4291b74031 Change deletes to preserve soft-deleted statuses in unresolved reports (#11805)
Change all account actions except "none" to resolve all unresolved reports

Refactor `SuspendAccountService` to be more readable
2019-09-11 16:32:44 +02:00
Eugen Rochko 8eab4349b4 Add retry for failed media downloads and `tootctl media refresh` (#11775) 2019-09-10 15:29:12 +02:00
Eugen Rochko 666c1a8a06 Change tootctl to use inline parallelization instead of Sidekiq (#11776)
- Remove --background option
- Add --concurrency(=5) option
- Add progress bars
2019-09-10 13:48:48 +02:00
Thibaut Girka eac7c66d2a Merge branch 'master' into glitch-soc/merge-upstream 2019-09-05 11:36:41 +02:00
Eugen Rochko 436266ca27 Change trending hashtags to not disappear instantly after midnight (#11712) 2019-09-02 18:11:13 +02:00
Thibaut Girka 449a0727a7 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/status.rb
- app/services/remove_status_service.rb
- db/schema.rb

All conflicts were due to the addition of a `deleted_at` attribute
to Statuses and reworked database indexes.
2019-08-29 12:07:50 +02:00
Eugen Rochko 32c781ec6b Add soft delete for statuses for instant deletes through API (#11623)
* Add soft delete for statuses to allow them to appear instant

* Allow reporting soft-deleted statuses and show them in the admin UI

* Change index for getting an account's statuses
2019-08-22 21:55:56 +02:00
Eugen Rochko e4605bbb4a Fix remote and staff-removed statuses leaving media behind for a day (#11638)
The reason for unattaching media instead of removing it is to support
delete & redraft functionality, but remote or staff-removed statuses
will never be redrafted, so the media should be deleted immediately
2019-08-22 04:17:12 +02:00
Thibaut Girka 81c6f2fd0e Merge branch 'master' into glitch-soc/merge-upstream 2019-08-08 08:44:33 +02:00
ThibG 072e696aa1 Add domain block notes (#11515)
* Add database columns for adding notes to domain blocks/restrctions

* Add admin UI to set private and public comments when blocking a domain

* Add text for private and public comments on domain blocks

* Show domain block comments in admin UI

* Add comments to the domain block undo page

* Make UnblockDomainService more robust regarding upgraded domain blocks

* Allow editing domain blocks

* Rename button from “undo domain block” to “view domain block” in account admin UI

* Change test to unsilence silenced users from upgraded blocks
2019-08-07 20:20:23 +02:00
Thibaut Girka 6c7a3268c9 Merge branch 'master' into glitch-soc/merge-upstream 2019-07-28 16:28:05 +02:00
mayaeh 7d13da568e Add `tootctl preview_cards remove` (#11320)
* Add `tootctl preview_cards remove`

* fix code style

* Remove `Scheduler::PreviewCardsCleanupScheduler` file

* fix code style again
Add exclude case where image_file_name is blank

* Added a function to output confirmation if the specified number of days is less than 2 weeks
2019-07-28 13:48:19 +02:00
Thibaut Girka 579ccb0bb9 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/application_controller.rb
- app/controllers/auth/confirmations_controller.rb
- app/controllers/auth/sessions_controller.rb
- app/controllers/settings/deletes_controller.rb
- app/controllers/settings/two_factor_authentication/recovery_codes_controller.rb
2019-07-23 10:51:07 +02:00
Eugen Rochko f9ed4354b1 Remove WebSub subscriptions (#11303) 2019-07-21 04:08:00 +02:00
Thibaut Girka eecce7e59b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- Gemfile.lock
- app/controllers/accounts_controller.rb
- app/controllers/admin/dashboard_controller.rb
- app/controllers/follower_accounts_controller.rb
- app/controllers/following_accounts_controller.rb
- app/controllers/remote_follow_controller.rb
- app/controllers/stream_entries_controller.rb
- app/controllers/tags_controller.rb
- app/javascript/packs/public.js
- app/lib/sanitize_config.rb
- app/models/account.rb
- app/models/form/admin_settings.rb
- app/models/media_attachment.rb
- app/models/stream_entry.rb
- app/models/user.rb
- app/serializers/initial_state_serializer.rb
- app/services/batched_remove_status_service.rb
- app/services/post_status_service.rb
- app/services/process_mentions_service.rb
- app/services/reblog_service.rb
- app/services/remove_status_service.rb
- app/views/admin/settings/edit.html.haml
- config/locales/simple_form.pl.yml
- config/settings.yml
- docker-compose.yml
2019-07-19 18:26:49 +02:00
Eugen Rochko ac853dca51 Add periodic removal of older thumbnails for preview cards (#11304) 2019-07-15 07:50:14 +02:00
han@highemelry eed19aa8be Change the retry limit in error of web push notification (#11292)
- Change the maximum count of retry for web push notification (Default -> 5).
   - In case of high load of subscribe server, the retries will be repeated many times.
   - Because the retries occupy the default queue, maximum retry count should be reduced.
2019-07-15 00:48:50 +02:00
han@highemelry b6997d2622 Change the retry limit in error of web push notification (#11292)
- Change the maximum count of retry for web push notification (Default -> 5).
   - In case of high load of subscribe server, the retries will be repeated many times.
   - Because the retries occupy the default queue, maximum retry count should be reduced.
2019-07-12 18:46:21 +02:00
Eugen Rochko 6baf5099a6 Refactor fetching of remote resources (#11251) 2019-07-10 18:59:28 +02:00