mastodon/db/post_migrate
Claire 02851848e9
Revamp post filtering system (#18058)
* Add model for custom filter keywords

* Use CustomFilterKeyword internally

Does not change the API

* Fix /filters/edit and /filters/new

* Add migration tests

* Remove whole_word column from custom_filters (covered by custom_filter_keywords)

* Redesign /filters

Instead of a list, present a card that displays more information and handles
multiple keywords per filter.

* Redesign /filters/new and /filters/edit to add and remove keywords

This adds a new gem dependency: cocoon, as well as a npm dependency:
cocoon-js-vanilla. Those are used to easily populate and remove form fields
from the user interface when manipulating multiple keyword filters at once.

* Add /api/v2/filters to edit filter with multiple keywords

Entities:
- `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context`
  `keywords`
- `FilterKeyword`: `id`, `keyword`, `whole_word`

API endpoits:
- `GET /api/v2/filters` to list filters (including keywords)
- `POST /api/v2/filters` to create a new filter
  `keywords_attributes` can also be passed to create keywords in one request
- `GET /api/v2/filters/:id` to read a particular filter
- `PUT /api/v2/filters/:id` to update a new filter
  `keywords_attributes` can also be passed to edit, delete or add keywords in
   one request
- `DELETE /api/v2/filters/:id` to delete a particular filter
- `GET /api/v2/filters/:id/keywords` to list keywords for a filter
- `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a
   filter
- `GET /api/v2/filter_keywords/:id` to read a particular keyword
- `PUT /api/v2/filter_keywords/:id` to edit a particular keyword
- `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword

* Change from `irreversible` boolean to `action` enum

* Remove irrelevent `irreversible_must_be_within_context` check

* Fix /filters/new and /filters/edit with update for filter_action

* Fix Rubocop/Codeclimate complaining about task names

* Refactor FeedManager#phrase_filtered?

This moves regexp building and filter caching to the `CustomFilter` class.

This does not change the functional behavior yet, but this changes how the
cache is built, doing per-custom_filter regexps so that filters can be matched
independently, while still offering caching.

* Perform server-side filtering and output result in REST API

* Fix numerous filters_changed events being sent when editing multiple keywords at once

* Add some tests

* Use the new API in the WebUI

- use client-side logic for filters we have fetched rules for.
  This is so that filter changes can be retroactively applied without
  reloading the UI.
- use server-side logic for filters we haven't fetched rules for yet
  (e.g. network error, or initial timeline loading)

* Minor optimizations and refactoring

* Perform server-side filtering on the streaming server

* Change the wording of filter action labels

* Fix issues pointed out by linter

* Change design of “Show anyway” link in accordence to review comments

* Drop “irreversible” filtering behavior

* Move /api/v2/filter_keywords to /api/v1/filters/keywords

* Rename `filter_results` attribute to `filtered`

* Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer

* Fix systemChannelId value in streaming server

* Simplify code by removing client-side filtering code

The simplifcation comes at a cost though: filters aren't retroactively
applied anymore.
2022-06-28 09:42:13 +02:00
..
.gitkeep Add post-deployment migration system (#8182) 2018-08-13 13:40:01 +02:00
20180813113448_copy_status_stats_cleanup.rb Move status counters to separate table, count replies (#8104) 2018-08-14 19:19:32 +02:00
20181116184611_copy_account_stats_cleanup.rb Extract counters from accounts table to account_stats table (#9295) 2018-11-19 00:43:52 +01:00
20190511152737_remove_suspended_silenced_account_fields.rb Record account suspend/silence time and keep track of domain blocks (#10660) 2019-05-14 19:05:02 +02:00
20190519130537_remove_boosts_widening_audience.rb Revert "Remove conversation URI (#11423)" (#11424) 2019-07-28 17:47:37 +02:00
20190706233204_drop_stream_entries.rb Remove Atom feeds and old URLs in the form of `GET /:username/updates/:id` (#11247) 2019-07-07 16:16:51 +02:00
20190715031050_drop_subscriptions.rb Remove WebSub subscriptions (#11303) 2019-07-21 04:08:00 +02:00
20190901040524_remove_score_from_tags.rb Change trending hashtags to not disappear instantly after midnight (#11712) 2019-09-02 18:11:13 +02:00
20190927124642_remove_invalid_web_push_subscription.rb Validate Web::PushSubscription (#11971) 2019-09-27 15:24:13 +02:00
20200917193528_migrate_notifications_type.rb Add option to be notified when a followed user posts (#13546) 2020-09-18 17:26:45 +02:00
20200917222734_remove_index_notifications_on_account_activity.rb Add option to be notified when a followed user posts (#13546) 2020-09-18 17:26:45 +02:00
20201017234926_fill_account_suspension_origin.rb Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
20210308133107_remove_subscription_expires_at_from_accounts.rb Remove subscription_expires_at leftover from OStatus (#15857) 2021-03-12 05:25:24 +01:00
20210502233513_drop_account_tag_stats.rb Change trending hashtags to be affected be reblogs (#16164) 2021-05-07 14:33:43 +02:00
20210507001928_remove_hub_url_from_accounts.rb Remove PubSubHubbub-related columns from accounts table (#16170) 2021-05-07 19:32:58 +02:00
20210526193025_remove_lock_version_from_account_stats.rb Fix account deletion sometimes failing because of optimistic locks (#16317) 2021-06-02 17:41:25 +02:00
20210616214135_remove_current_sign_in_ip_from_users.rb Remove IP tracking columns from users table (#16409) 2022-01-16 13:23:50 +01:00
20210808071221_clear_orphaned_account_notes.rb Fix owned account notes not being deleted when an account is deleted (#16579) 2021-08-08 15:29:57 +02:00
20211126000907_drop_account_identity_proofs.rb Remove Keybase integration (#17045) 2021-11-26 05:58:18 +01:00
20220109213908_remove_action_taken_from_reports.rb Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
20220118183010_remove_index_users_on_remember_token.rb Remove leftover database columns from Devise::Models::Rememberable (#17191) 2022-01-23 15:46:30 +01:00
20220118183123_remove_rememberable_from_users.rb Remove leftover database columns from Devise::Models::Rememberable (#17191) 2022-01-23 15:46:30 +01:00
20220202201015_remove_trust_level_from_accounts.rb Add trending statuses (#17431) 2022-02-25 00:34:14 +01:00
20220303203437_remove_media_attachments_changed_from_status_edits.rb Change how changes to media attachments are stored for edits (#17696) 2022-03-09 09:06:17 +01:00
20220307083603_optimize_null_index_conversations_uri.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060545_optimize_null_index_statuses_in_reply_to_account_id.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060556_optimize_null_index_statuses_in_reply_to_id.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060614_optimize_null_index_media_attachments_scheduled_status_id.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060626_optimize_null_index_media_attachments_shortcode.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060641_optimize_null_index_users_reset_password_token.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060653_optimize_null_index_users_created_by_application_id.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060706_optimize_null_index_statuses_uri.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060722_optimize_null_index_accounts_moved_to_account_id.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060740_optimize_null_index_oauth_access_tokens_refresh_token.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060750_optimize_null_index_accounts_url.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060809_optimize_null_index_oauth_access_tokens_resource_owner_id.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060833_optimize_null_index_announcement_reactions_custom_emoji_id.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060854_optimize_null_index_appeals_approved_by_account_id.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060913_optimize_null_index_account_migrations_target_account_id.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060926_optimize_null_index_appeals_rejected_by_account_id.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060939_optimize_null_index_list_accounts_follow_id.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220310060959_optimize_null_index_web_push_subscriptions_access_token_id.rb Fix null values being included in some indexes (#17711) 2022-03-12 08:12:57 +01:00
20220429101025_remove_ips_from_email_domain_blocks.rb Remove IP matching from e-mail domain blocks (#18190) 2022-04-29 23:27:03 +02:00
20220429101850_clear_email_domain_blocks.rb Remove IP matching from e-mail domain blocks (#18190) 2022-04-29 23:27:03 +02:00
20220527114923_remove_filtered_languages_from_users.rb Remove unused `filtered_languages` column (#18533) 2022-05-27 20:05:22 +02:00
20220613110802_remove_whole_word_from_custom_filters.rb Revamp post filtering system (#18058) 2022-06-28 09:42:13 +02:00
20220613110903_remove_irreversible_from_custom_filters.rb Revamp post filtering system (#18058) 2022-06-28 09:42:13 +02:00