mastodon/app/models
Kaspar V f2ec356cb6 refactor(vacuum statuses): reduce amount of db queries and load for each query - improve performance (#21487)
* refactor(statuses_vacuum): remove dead code - unused

Method is not called inside class and private.
Clean up dead code.

* refactor(statuses_vacuum): make retention_period present test explicit

This private method only hides functionality.
It is best practice to be as explicit as possible.

* refactor(statuses_vacuum): improve query performance

- fix statuses_scope having sub-select for Account.remote scope by
  `joins(:account).merge(Account.remote)`
- fix statuses_scope unnecessary use of `Status.arel_table[:id].lt`
  because it is inexplicit, bad practice and even slower than normal
  `.where('statuses.id < ?'`
- fix statuses_scope remove select(:id, :visibility) for having reusable
  active record query batches (no re queries)
- fix vacuum_statuses! to use in_batches instead of find_in_batches,
  because in_batches delivers a full blown active record query result,
  in stead of an array - no requeries necessary
- send(:unlink_from_conversations) not to perform another db query, but
  reuse the in_batches result instead.
- remove now obsolete remove_from_account_conversations method
- remove_from_search_index uses array of ids, instead of mapping
  the ids from an array - this should be more efficient
- use the in_batches scope to call delete_all, instead of running
  another db query for this - because it is again more efficient
- add TODO comment for calling models private method with send

* refactor(status): simplify unlink_from_conversations

- add `has_many through:` relation mentioned_accounts
- use model scope local instead of method call `Status#local?`
- more readable add account to inbox_owners when account.local?

* refactor(status): searchable_by way less sub selects

These queries all included a sub-select. Doing the same with a joins
should be more efficient.
Since this method does 5 such queries, this should be significant,
since it technically halves the query count.

This is how it was:

```ruby
[3] pry(main)> Status.first.mentions.where(account: Account.local, silent: false).explain
  Status Load (1.6ms)  SELECT "statuses".* FROM "statuses" WHERE "statuses"."deleted_at" IS NULL ORDER BY "statuses"."id" DESC LIMIT $1  [["LIMIT", 1]]
  Mention Load (1.5ms)  SELECT "mentions".* FROM "mentions" WHERE "mentions"."status_id" = $1 AND "mentions"."account_id" IN (SELECT "accounts"."id" FROM "accounts" WHERE "accounts"."domain" IS NULL) AND "mentions"."silent" = $2  [["status_id", 109382923142288414], ["silent", false]]
=> EXPLAIN for: SELECT "mentions".* FROM "mentions" WHERE "mentions"."status_id" = $1 AND "mentions"."account_id" IN (SELECT "accounts"."id" FROM "accounts" WHERE "accounts"."domain" IS NULL) AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
                                                    QUERY PLAN
------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.15..23.08 rows=1 width=41)
   ->  Seq Scan on accounts  (cost=0.00..10.90 rows=1 width=8)
         Filter: (domain IS NULL)
   ->  Index Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.15..8.17 rows=1 width=41)
         Index Cond: ((account_id = accounts.id) AND (status_id = '109382923142288414'::bigint))
         Filter: (NOT silent)
(6 rows)
```

This is how it is with this change:

```ruby
[4] pry(main)> Status.first.mentions.joins(:account).merge(Account.local).active.explain
  Status Load (1.7ms)  SELECT "statuses".* FROM "statuses" WHERE "statuses"."deleted_at" IS NULL ORDER BY "statuses"."id" DESC LIMIT $1  [["LIMIT", 1]]
  Mention Load (0.7ms)  SELECT "mentions".* FROM "mentions" INNER JOIN "accounts" ON "accounts"."id" = "mentions"."account_id" WHERE "mentions"."status_id" = $1 AND "accounts"."domain" IS NULL AND "mentions"."silent" = $2  [["status_id", 109382923142288414], ["silent", false]]
=> EXPLAIN for: SELECT "mentions".* FROM "mentions" INNER JOIN "accounts" ON "accounts"."id" = "mentions"."account_id" WHERE "mentions"."status_id" = $1 AND "accounts"."domain" IS NULL AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
                                                    QUERY PLAN
------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.15..23.08 rows=1 width=41)
   ->  Seq Scan on accounts  (cost=0.00..10.90 rows=1 width=8)
         Filter: (domain IS NULL)
   ->  Index Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.15..8.17 rows=1 width=41)
         Index Cond: ((account_id = accounts.id) AND (status_id = '109382923142288414'::bigint))
         Filter: (NOT silent)
(6 rows)
```
2022-11-27 20:41:18 +01:00
..
account Don't allow URLs that contain non-normalized paths to be verified (#20999) 2022-11-20 19:28:13 +01:00
account_suggestions Fix single Redis connection being used across all threads (#18135) 2022-04-28 17:47:34 +02:00
admin Support for import/export of instance-level domain blocks/allows for 4.x w/ additional fixes (#20597) 2022-11-17 11:05:09 +01:00
concerns Fix error when invalid domain name is submitted (#19474) 2022-11-14 08:07:14 +01:00
form Support for import/export of instance-level domain blocks/allows for 4.x w/ additional fixes (#20597) 2022-11-17 11:05:09 +01:00
trends Fix error when passing unknown filter param in REST API (#20626) 2022-11-14 08:06:06 +01:00
web Add `policy` param to `POST /api/v1/push/subscriptions` (#16040) 2021-04-15 05:00:25 +02:00
account.rb Fix trendable status without review (#20214) 2022-11-11 21:24:10 +01:00
account_alias.rb Micro-optimization: only split `acct` into two Strings (#19901) 2022-11-07 16:17:55 +01:00
account_conversation.rb Fix single Redis connection being used across all threads (#18135) 2022-04-28 17:47:34 +02:00
account_deletion_request.rb Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
account_domain_block.rb
account_filter.rb Fix error when passing unknown filter param in REST API (#20626) 2022-11-14 08:06:06 +01:00
account_migration.rb Skip Webfinger cache during migrations as well (#19883) 2022-11-07 03:31:38 +01:00
account_moderation_note.rb
account_note.rb Fix AccountNote not having a maximum length (#16942) 2021-11-06 00:12:25 +01:00
account_pin.rb
account_stat.rb Fix follower and other counters being able to go negative (#18517) 2022-05-26 20:32:48 +02:00
account_statuses_cleanup_policy.rb Micro-optimization: use `if`/`else` instead of `Array#compact` and `Array#min` (#19906) 2022-11-08 03:50:47 +01:00
account_statuses_filter.rb Fix performance of account timelines (#17709) 2022-03-08 09:14:39 +01:00
account_suggestions.rb Change auto-following admin-selected accounts, show in recommendations (#16078) 2021-04-24 17:01:43 +02:00
account_summary.rb Fix FollowRecommendationsScheduler failing because of unpopulated views (#16189) 2021-05-09 10:39:29 +02:00
account_warning.rb Add audit log entries for user roles (#19040) 2022-08-25 20:39:40 +02:00
account_warning_preset.rb Add titles to warning presets in admin UI (#13252) 2020-03-12 17:57:59 +01:00
admin.rb
announcement.rb Change admin announcement edition interface to use datetime-local (#18321) 2022-10-28 12:56:32 +02:00
announcement_filter.rb Fix error when passing unknown filter param in REST API (#20626) 2022-11-14 08:06:06 +01:00
announcement_mute.rb Add announcements (#12662) 2020-01-23 22:00:13 +01:00
announcement_reaction.rb Add announcements (#12662) 2020-01-23 22:00:13 +01:00
appeal.rb Add audit log entries for user roles (#19040) 2022-08-25 20:39:40 +02:00
application_record.rb Fix records not being indexed sometimes (#12024) 2019-10-01 01:19:11 +02:00
backup.rb Add announcements (#12662) 2020-01-23 22:00:13 +01:00
block.rb
bookmark.rb Bump chewy from 5.2.0 to 7.2.3 (supports Elasticsearch 7.x) (#16915) 2021-11-18 22:02:08 +01:00
canonical_email_block.rb Add admin API for managing canonical e-mail blocks (#19067) 2022-08-28 03:31:54 +02:00
content_retention_policy.rb Add retention policy for cached content and media (#19232) 2022-09-27 03:08:19 +02:00
context.rb
conversation.rb
conversation_mute.rb
custom_emoji.rb Fix metadata scrubbing removing color profile from images (#20389) 2022-11-11 09:20:10 +01:00
custom_emoji_category.rb
custom_emoji_filter.rb Fix error when passing unknown filter param in REST API (#20626) 2022-11-14 08:06:06 +01:00
custom_filter.rb Add ability to filter individual posts (#18945) 2022-08-25 04:27:47 +02:00
custom_filter_keyword.rb Change how hashtags are normalized (#18795) 2022-07-13 15:03:28 +02:00
custom_filter_status.rb Add ability to select all accounts matching search for batch actions (#19053) 2022-08-25 23:33:34 +02:00
device.rb Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
domain_allow.rb Support for import/export of instance-level domain blocks/allows for 4.x w/ additional fixes (#20597) 2022-11-17 11:05:09 +01:00
domain_block.rb Support for import/export of instance-level domain blocks/allows for 4.x w/ additional fixes (#20597) 2022-11-17 11:05:09 +01:00
email_domain_block.rb Add admin API for managing e-mail domain blocks (#19066) 2022-08-28 03:37:55 +02:00
encrypted_message.rb Fix single Redis connection being used across all threads (#18135) 2022-04-28 17:47:34 +02:00
export.rb Add ability to filter followed accounts' posts by language (#19095) 2022-09-20 23:51:21 +02:00
extended_description.rb Change about page to be mounted in the web UI (#19345) 2022-10-13 14:42:37 +02:00
favourite.rb Bump chewy from 5.2.0 to 7.2.3 (supports Elasticsearch 7.x) (#16915) 2021-11-18 22:02:08 +01:00
featured_tag.rb Improve performance by avoiding regex construction (#20215) 2022-11-10 05:49:30 +01:00
feed.rb allow pagination by min_id and max_id (#14776) 2020-09-12 17:09:49 +02:00
follow.rb Add ability to filter followed accounts' posts by language (#19095) 2022-09-20 23:51:21 +02:00
follow_recommendation.rb Fix FollowRecommendationsScheduler failing because of unpopulated views (#16189) 2021-05-09 10:39:29 +02:00
follow_recommendation_filter.rb Fix single Redis connection being used across all threads (#18135) 2022-04-28 17:47:34 +02:00
follow_recommendation_suppression.rb Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
follow_request.rb Add ability to filter followed accounts' posts by language (#19095) 2022-09-20 23:51:21 +02:00
home_feed.rb Fix rubocop config and warnings (#15503) 2021-01-07 09:40:55 +01:00
identity.rb
import.rb Fix follow limit preventing re-following of a moved account (#14207) 2020-12-18 09:18:31 +01:00
instance.rb Add audit log entries for user roles (#19040) 2022-08-25 20:39:40 +02:00
instance_filter.rb Fix error when passing unknown filter param in REST API (#20626) 2022-11-14 08:06:06 +01:00
invite.rb Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
invite_filter.rb Fix error when passing unknown filter param in REST API (#20626) 2022-11-14 08:06:06 +01:00
ip_block.rb Fix IP blocks not having a unique index (#19456) 2022-10-25 21:43:44 +02:00
list.rb Improve account deletion performances further (#15407) 2020-12-22 23:57:46 +01:00
list_account.rb Add abilityto add oneself to lists (#12271) 2019-11-04 13:02:01 +01:00
list_feed.rb Fix rubocop config and warnings (#15503) 2021-01-07 09:40:55 +01:00
login_activity.rb Add authentication history (#16408) 2021-06-21 17:07:30 +02:00
marker.rb
media_attachment.rb fix media uploads with ffmpeg 5 (#21191) 2022-11-25 16:20:47 +01:00
mention.rb
message_franking.rb Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
mute.rb Add duration parameter to muting. (#13831) 2020-10-13 01:01:14 +02:00
notification.rb Add notifications for new reports (#18697) 2022-06-27 09:30:15 +02:00
one_time_key.rb Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
poll.rb Clear voter count when poll is reset (#21700) 2022-11-26 23:08:25 +01:00
poll_vote.rb
preview_card.rb Fix metadata scrubbing removing color profile from images (#20389) 2022-11-11 09:20:10 +01:00
preview_card_provider.rb Fix metadata scrubbing removing color profile from images (#20389) 2022-11-11 09:20:10 +01:00
preview_card_trend.rb Add support for language preferences for trending statuses and links (#18288) 2022-10-08 16:45:40 +02:00
privacy_policy.rb Fix privacy policy being empty if custom setting exists but is empty (#19318) 2022-10-08 08:34:00 +02:00
public_feed.rb Revert filtering public timelines by locale by default (#20294) 2022-11-10 05:34:42 +01:00
relationship_filter.rb Fix error when passing unknown filter param in REST API (#20626) 2022-11-14 08:06:06 +01:00
relay.rb Fix enable/disable relay failures (#13535) 2020-04-23 22:04:18 +02:00
remote_follow.rb Remove dependency on goldfinger gem (#14919) 2020-10-08 00:34:57 +02:00
report.rb Fix notifications about deleted reports not being also deleted (#19475) 2022-10-27 02:10:54 +02:00
report_filter.rb Fix error when passing unknown filter param in REST API (#20626) 2022-11-14 08:06:06 +01:00
report_note.rb
rule.rb Fix rules with same priority being sorted non-deterministically (#20623) 2022-11-14 06:28:19 +01:00
scheduled_status.rb
search.rb
session_activation.rb Add Ruby 3.0 support (#16046) 2021-05-06 14:22:54 +02:00
setting.rb Use Rails' index_by where it makes sense (#15542) 2021-01-12 09:27:38 +01:00
site_upload.rb Fix metadata scrubbing removing color profile from images (#20389) 2022-11-11 09:20:10 +01:00
status.rb refactor(vacuum statuses): reduce amount of db queries and load for each query - improve performance (#21487) 2022-11-27 20:41:18 +01:00
status_edit.rb Add ability to view previous edits of a status in admin UI (#19462) 2022-10-26 13:42:29 +02:00
status_pin.rb Add feature to automatically delete old toots (#16529) 2021-08-09 23:11:50 +02:00
status_stat.rb Fix follower and other counters being able to go negative (#18517) 2022-05-26 20:32:48 +02:00
status_trend.rb Fix trending statuses returning more than one post by the same author (#19349) 2022-10-14 01:44:23 +02:00
system_key.rb Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
tag.rb Improve performance by avoiding regex construction (#20215) 2022-11-10 05:49:30 +01:00
tag_feed.rb Revert filtering public timelines by locale by default (#20294) 2022-11-10 05:34:42 +01:00
tag_follow.rb Add ability to follow hashtags (#18809) 2022-07-17 13:49:29 +02:00
tombstone.rb
trends.rb Fix missing `skip_review?` (#19335) 2022-10-10 08:03:19 +02:00
unavailable_domain.rb Add audit log entries for user roles (#19040) 2022-08-25 20:39:40 +02:00
user.rb Change automatic post deletion configuration to be accessible to redirected users (#20774) 2022-11-17 10:55:23 +01:00
user_invite_request.rb
user_ip.rb Remove IP tracking columns from users table (#16409) 2022-01-16 13:23:50 +01:00
user_role.rb Add audit log entries for user roles (#19040) 2022-08-25 20:39:40 +02:00
web.rb Add extended about page stub 2017-01-13 03:24:41 +01:00
webauthn_credential.rb Fix validates :sign_count of WebauthnCredential (#14806) 2020-09-16 20:16:46 +02:00
webhook.rb Add administrative webhooks (#18510) 2022-06-09 21:57:36 +02:00