Commit Graph

1322 Commits (22cdd53335cae33b54ebede4bb1464c6189806f8)

Author SHA1 Message Date
Claire 8114f4208f Remove leftover database columns from Devise::Models::Rememberable (#17191)
* Remove leftover database columns from Devise::Models::Rememberable

* Update fix-duplication maintenance script

* Improve errors/warnings in the fix-duplicates maintenance script
2022-01-23 15:46:30 +01:00
Eugen Rochko 06b698a723 Add support for editing for published statuses (#16697)
* Add support for editing for published statuses

* Fix references to stripped-out code

* Various fixes and improvements

* Further fixes and improvements

* Fix updates being potentially sent to unauthorized recipients

* Various fixes and improvements

* Fix wrong words in test

* Fix notifying accounts that were tagged but were not in the audience

* Fix mistake
2022-01-19 22:37:27 +01:00
Eugen Rochko 2fb76550a9 Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
Eugen Rochko 7b3972c270 Remove IP tracking columns from users table (#16409) 2022-01-16 13:23:50 +01:00
Jeong Arm ba93eb818b Fix duplicate record on admin/accounts when searching with IP (#17150) 2021-12-21 00:17:14 +01:00
Claire d8da1689c1 Add ability for admins to delete canonical email blocks (#16644)
* Add admin option to remove canonical email blocks from a deleted account

* Add tootctl canonical_email_blocks to inspect and remove canonical email blocks
2021-12-17 23:02:14 +01:00
Claire 44c8b6ff2f Add ability to purge undeliverable domains from admin interface (#16686)
* Add ability to purge undeliverable domains from admin interface

* Add tests
2021-12-17 23:01:21 +01:00
Eugen Rochko 2c6be5dc9a Change trending hashtags threshold back from 15 to 5 (#17122) 2021-12-13 05:32:29 +01:00
Eugen Rochko 7bf5924339 Add batch suspend for accounts in admin UI (#17009) 2021-12-05 21:48:39 +01:00
Eugen Rochko 4375813ea7 Remove Keybase integration (#17045) 2021-11-26 05:58:18 +01:00
Eugen Rochko 6c1d78b277 Fix error on trending hashtags/links pages in admin UI due to missing constant (#17044) 2021-11-26 01:12:39 +01:00
Eugen Rochko ad73becf3e Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire 5d2ed78074 Fix error when suspending user with an already-existing canonical email block (#17036)
* Fix error when suspending user with an already-existing canonical email block

Fixes #17033

While attempting to create a `CanonicalEmailBlock` with an existing hash would
raise an `ActiveRecord::RecordNotUnique` error, this being done within a
transaction would cancel the whole transaction. For this reason, checking for
uniqueness in Rails would query the database within the transaction and avoid
invalidating the whole transaction for this reason.

A race condition is still possible, where multiple accounts sharing a canonical
email would be blocked in concurrent transactions, in which only one would
succeed, but that is way less likely to happen that the current issue, and can
always be retried after the first failure, unlike the current situation.

* Add tests
2021-11-24 17:41:03 +01:00
Takeshi Umeda 869faf5f87 Bump chewy from 5.2.0 to 7.2.3 (supports Elasticsearch 7.x) (#16915)
* Bump chewy from 5.2.0 to 7.2.2

* fix style (codeclimate)

* fix style

* fix style

* Bump chewy from 7.2.2 to 7.2.3
2021-11-18 22:02:08 +01:00
Claire 9e122d774d Fix reviving revoked sessions and invalidating login (#16943)
Up until now, we have used Devise's Rememberable mechanism to re-log users
after the end of their browser sessions. This mechanism relies on a signed
cookie containing a token. That token was stored on the user's record,
meaning it was shared across all logged in browsers, meaning truly revoking
a browser's ability to auto-log-in involves revoking the token itself, and
revoking access from *all* logged-in browsers.

We had a session mechanism that dynamically checks whether a user's session
has been disabled, and would log out the user if so. However, this would only
clear a session being actively used, and a new one could be respawned with
the `remember_user_token` cookie.

In practice, this caused two issues:
- sessions could be revived after being closed from /auth/edit (security issue)
- auto-log-in would be disabled for *all* browsers after logging out from one
  of them

This PR removes the `remember_token` mechanism and treats the `_session_id`
cookie/token as a browser-specific `remember_token`, fixing both issues.
2021-11-06 00:13:58 +01:00
Claire 925adbf7af Fix AccountNote not having a maximum length (#16942) 2021-11-06 00:12:25 +01:00
Claire 91bd8b921b Fix handling announcements with links (#16941)
Broken since #15827
2021-11-05 21:14:35 +01:00
Takeshi Umeda dae4762733 Add remove from followers api (#16864)
* Add followed_by? to account_interactions

* Add RemoveFromFollowersService

* Fix AccountBatch to use RemoveFromFollowersService

* Add remove from followers API
2021-10-18 12:02:35 +02:00
Jeong Arm 95688e49b7 Allow keeping only 1 boosts/favs on auto deleting posts (#16653)
* Allow keeping 1 boosts/favs on auto deleting posts

* Fix tests
2021-10-14 21:11:14 +02:00
Claire 7cc72ace9f Fix 404 error when filtering admin action logs by non-existent target account (#16643)
Currently, there is no way for an admin to delete an account record. However,
should that happen in the future, or should an admin mistype an account id,
`/admin/action_logs?target_account_id=<non-existent-account-id>` currently
returns a 404 instead of an empty list.

With this commit, the aforementioned page returns a correct list instead
(potentially non-empty if the account has been manually deleted)
2021-10-14 20:53:11 +02:00
Claire 4a813ec686 Fix scheduled statuses decreasing statuses counts (#16791)
* Add tests

* Fix scheduled statuses decreasing statuses counts

Fixes #16774
2021-10-14 19:59:21 +02:00
Claire 879229e098 Fix error when rendering public pages with media attachments (#16763)
* Add tests

* Fix error when rendering public pages with media attachments

* Add tests

* Fix tests

* Please CodeClimate
2021-10-13 15:27:19 +02:00
Claire 10918d97d4 Fix media attachment size validation not correctly accounting for file type (#16819)
* Fix media attachment size validation not correctly accounting for file type

Fixes a regression introduced in #16724 caused by the fact that kt-paperclip
now correctly runs validations before processing, meaning that file size
verification could not rely on our before_post_processing hook.

Moved the `before_post_processing` hooks to `before_validate` to make sure
the media attachment type is set correctly before the file gets validated.

* Add tests
2021-10-06 14:49:32 +01:00
Claire 7d0089033f Switch from unmaintained paperclip to kt-paperclip (#16724)
* Switch from unmaintained paperclip to kt-paperclip

* Drop some compatibility monkey-patches not required by kt-paperclip

* Drop media spoof check monkey-patching

It's broken with kt-paperclip and hopefully it won't be needed anymore

* Fix regression introduced by paperclip 6.1.0

* Do not rely on pathname to call FastImage

* Add test for ogg vorbis file with cover art

* Add audio/vorbis to the accepted content-types

This seems erroneous as this would be the content-type for a vorbis stream
without an ogg container, but that's what the `marcel` gem outputs, so…

* Restore missing for_as_default method

* Refactor Attachmentable concern and delay Paperclip's content-type spoof check

Check for content-type spoofing *after* setting the extension ourselves, this
fixes a regression with kt-paperclip's validations being more strict than
paperclip 6.0.0 and rejecting some Pleroma uploads because of unknown
extensions.

* Please CodeClimate

* Add audio/vorbis to the unreliable set

It doesn't correspond to a file format and thus has no extension associated.
2021-09-29 23:52:36 +02:00
Claire 4080a8ec33 Stop setting a shortcode to newly-created media attachments (#16730)
* Stop setting a shortcode to newly-created media attachments

The WebUI has stopped using the “short media URL” in ages. This isn't used
anywhere except for mail notifications.

Deprecating it would allow us to eventually get rid of at least a database
column and corruption-prone index, as well as a controller.

* Fix tests
2021-09-13 18:59:37 +02:00
Claire a20cb503ec Fix processing mentions to domains with non-ascii TLDs (#16689)
Fixes #16602
2021-09-01 22:06:40 +02:00
Claire 1692e0b381 Fix followers synchronization mechanism not working when URI has empty path (#16510)
* Fix followers synchronization mechanism not working when URI has empty path

To my knowledge, there is no current implementation on the fediverse
that can use bare domains (e.g., actor is at https://example.org instead of
something like https://example.org/actor) that also plans to support the
followers synchronization mechanism. However, Mastodon's current implementation
would exclude such accounts from followers list.

Also adds tests and rename them to reflect the proper method names.

* Move url prefix regexp to its own constant
2021-08-11 17:48:42 +02:00
Claire acf7595157 Fix crash when encountering invalid account fields (#16598)
* Add test

* Fix crash when encountering invalid account fields
2021-08-11 16:40:55 +02:00
Claire 1fcf310c60 Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Claire 17318962de Fix owned account notes not being deleted when an account is deleted (#16579)
* Add account_notes relationship

* Add tests

* Fix owned account notes not being deleted when an account is deleted

* Add post-migration to clean up orphaned account notes
2021-08-08 15:29:57 +02:00
Claire 1e452a10b8 Fix user email address being banned on self-deletion (#16503)
* Add tests

* Fix user email address being banned on self-deletion

Fixes #16498
2021-07-14 05:35:49 +02:00
Eugen Rochko 2bc9e256cc Add ability to filter reports by origin of target account (#16487) 2021-07-11 11:01:38 +02:00
Eugen Rochko 34dd743b58 Add ability to skip sign-in token authentication for specific users (#16427)
Remove "active within last two weeks" exception for sign in token requirement

Change admin reset password to lock access until the password is reset
2021-07-08 05:31:28 +02:00
Claire d40dedaf93 Fix upgrade path from 3.4.0 (#16465)
3.4.1 dropped account_stats.lock_version, but in a way breaking the usual
upgrade path by requiring services to be reloaded after the post-migrations.

Indeed, `self.locking_column = nil` was not enough for Rails to ignore the
`lock_version` column when preparing statements on application load, resulting
in some ActiveRecord queries (typically those involving
`includes(:account_stat)`) erroring out with:

  ActiveRecord::StatementInvalid (PG::UndefinedColumn: ERROR:  column account_stats.lock_version does not exist
2021-07-07 21:13:30 +02:00
Eugen Rochko 681ebba604 Fix not being able to suspend accounts that already have a canonical e-mail block (#16455) 2021-07-02 12:03:15 +02:00
Eugen Rochko 2067b0bf34 Add authentication history (#16408) 2021-06-21 17:07:30 +02:00
Claire 2c3a1f32ad Fix account deletion sometimes failing because of optimistic locks (#16317)
* Fix account deletion sometimes failing because of optimistic locks

In some rare occasions[1], deleting accounts would fail with a
`StaleObjectError` exception.

Indeed, account deletion manually sets the `AccountStat` values without
handling cases where the optimistic locking on `AccountStat` would fail.

To my knowledge, with the rewrite of account counters in #15913, the
`DeleteAccountService` is now the only place that changes the counters in
a way that is not atomic.

Since in this specific case, we do not care about the previous values of the
account counters, it appears we don't need locking at all for this table
anymore.

[1]: https://discourse.joinmastodon.org/t/account-cant-be-deleted/3602

* Bump MAX_SUPPORTED_VERSION in maintenance script
2021-06-02 17:41:25 +02:00
Claire 2934b06a25 Update fix-duplicates maintenance script to support latest migrations (#16231)
* Update maintenance script to support latest database migrations

* Update Account#merge_with!
2021-05-12 23:19:44 +02:00
Takeshi Umeda 43eeefdfcf Fix to be able to redownload avatar and header (#16190)
* Fix to reset if header and avatar download fails

* Add RedownloadAvatarWorker and RedownloadHeaderWorker
2021-05-11 14:19:22 +02:00
Claire 4164a7bfe8 Fix FollowRecommendationsScheduler failing because of unpopulated views (#16189)
Since #16173, `account_summaries` and `follow_recommendations` are not
populated at creation time, which causes concurrent refresh to fail.

As we currently only use those materialized views right after explicitly
refreshing them, this commit changes refreshes to not be performed
concurrently. This will fix the issue and ensure the refresh completes
faster while using less resources.
2021-05-09 10:39:29 +02:00
Eugen Rochko 909c524fa7 Remove PubSubHubbub-related columns from accounts table (#16170) 2021-05-07 19:32:58 +02:00
Eugen Rochko 91819606f9 Change trending hashtags to be affected be reblogs (#16164)
If a status with a hashtag becomes very popular, it stands to
reason that the hashtag should have a chance at trending

Fix no stats being recorded for hashtags that are not allowed
to trend, and stop ignoring bots

Remove references to hashtags in profile directory from the code
and the admin UI
2021-05-07 14:33:43 +02:00
Claire 86f5fad111 Add Ruby 3.0 support (#16046)
* Fix issues with POSIX::Spawn, Terrapin and Ruby 3.0

Also improve the Terrapin monkey-patch for the stderr/stdout issue.

* Fix keyword argument handling throughout the codebase

* Monkey-patch Paperclip to fix keyword arguments handling in validators

* Change validation_extensions to please CodeClimate

* Bump microformats from 4.2.1 to 4.3.1

* Allow Ruby 3.0

* Add Ruby 3.0 test target to CircleCI

* Add test for admin dashboard warnings

* Fix admin dashboard warnings on Ruby 3.0
2021-05-06 14:22:54 +02:00
Takeshi Umeda c958bc7297 Add management of delivery availability in Federation settings (#15771)
* Add management of delivery availavility in Federation settings

* fix translate

* Remove useless object creation

* Fix DeepSource issue

* Add shortcut for all

* Fix DeepSource(skipcq)

* Change 'remove' to 'clear'

* Fix style

* Change class method name (exhausted_deliveries_key_by)
2021-05-05 23:39:02 +02:00
Claire 78eddfc83c Improve performance of follow recommendation scheduler (#16159)
Express follow_recommendations in terms of account_summaries rather than
accounts, integrate filters that are unconditionally used, and materialize
the resulting view.

This should result in the bulk of the computation being performed only once
instead of **once per recommendation language**.
2021-05-05 22:04:52 +02:00
Eugen Rochko 1c2b085a64 Fix media processing getting stuck on too much stdin/stderr (#16136)
* Fix media processing getting stuck on too much stdin/stderr

See thoughtbot/terrapin#5

* Remove dependency on paperclip-av-transcoder gem

* Remove dependency on streamio-ffmpeg gem

* Disable stdin on ffmpeg process
2021-05-05 19:44:01 +02:00
Eugen Rochko 876061e777 Fix empty home feed before first follow has finished processing (#16152)
Change queue of merge worker from pull to default
2021-05-04 04:45:08 +02:00
abcang fd2c5e27c7 Improve tag search query (#16104) 2021-04-25 06:33:28 +02:00
Eugen Rochko f5d59b3979 Change auto-following admin-selected accounts, show in recommendations (#16078) 2021-04-24 17:01:43 +02:00
Eugen Rochko bf5d831fe7 Fix nil error when removing status caused by race condition (#16099) 2021-04-24 13:35:39 +02:00
Claire 3b3907ec9d Fix edge case where accepted follow cannot be processed because of follow limit (#16098) 2021-04-23 22:51:21 +02:00
Eugen Rochko c9bb0e576d Add canonical e-mail blocks for suspended accounts (#16049)
Prevent new accounts from being created using the same underlying
e-mail as a suspended account using extensions and period
permutations. Stores e-mails as a SHA256 hash
2021-04-17 03:14:25 +02:00
Eugen Rochko 2cab9c9f06 Add `policy` param to `POST /api/v1/push/subscriptions` (#16040)
With possible values `all`, `followed`, `follower`, and `none`,
control from whom notifications will generate a Web Push alert
2021-04-15 05:00:25 +02:00
Eugen Rochko a8bf951926 Change Web Push API deliveries to use request pooling (#16014) 2021-04-12 14:25:34 +02:00
Eugen Rochko 92b2d926bf Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
Eugen Rochko 2e8a673376 Remove spam check and dependency on nilsimsa gem (#16011) 2021-04-11 11:22:50 +02:00
Claire b2a89bf38e Update Mastodon to Rails 6.1 (#15910)
* Update devise-two-factor to unreleased fork for Rails 6 support

Update tests to match new `rotp` version.

* Update nsa gem to unreleased fork for Rails 6 support

* Update rails to 6.1.3 and rails-i18n to 6.0

* Update to unreleased fork of pluck_each for Ruby 6 support

* Run "rails app:update"

* Add missing ActiveStorage config file

* Use config.ssl_options instead of removed ApplicationController#force_ssl

Disabled force_ssl-related tests as they do not seem to be easily testable
anymore.

* Fix nonce directives by removing Rails 5 specific monkey-patching

* Fix fixture_file_upload deprecation warning

* Fix yield-based test failing with Rails 6

* Use Rails 6's index_with when possible

* Use ActiveRecord::Cache::Store#delete_multi from Rails 6

This will yield better performances when deleting an account

* Disable Rails 6.1's automatic preload link headers

Since Rails 6.1, ActionView adds preload links for javascript files
in the Links header per default.

In our case, that will bloat headers too much and potentially cause
issues with reverse proxies. Furhermore, we don't need those links,
as we already output them as HTML link tags.

* Switch to Rails 6.0 default config

* Switch to Rails 6.1 default config

* Do not include autoload paths in the load path
2021-03-24 10:44:31 +01:00
Eugen Rochko 3c1b807b94 WIP (#15222) 2021-03-19 17:15:36 +01:00
Claire 5c69eecacc Improve account counters handling (#15913)
* Improve account counters handling

* Use ActiveRecord::Base::sanitize_sql to pass values instead of interpolating them

Keep using string interpolation for `key` as it is safe and using
“ActiveRecord::Base::sanitize_sql_hash_for_assignment” would require stitching
bits of SQL in a way that is not more easily checked for safety.

* Add migration hook to catch PostgreSQL versions earlier than 9.5
2021-03-19 13:14:57 +01:00
Claire 1634e71bd6 Prepare Mastodon for Rails 6 (#15911)
* Fix misuse of foreign_type

* Fix use of removed "add_template_helper"

* Use response.media_type instead of response.content_type in tests

* Fix CSV export controller test on Rails 6

Rails 6 sets a "filename*" field in the Content-Disposition header to
explicitly encode the filename as UTF-8.

This changes checks the first part of the Content-Disposition header so
it matches in both Rails 5 and Rails 6.

* Fix emoji formatting with Rails 6

* Make emoji output more idiomatic and robust

* Switch from redis-rails gem to built-in Rails redis cache storage
2021-03-17 10:09:55 +01:00
Claire f05b0ab58a Fix not being able to change world filter expiration back to “Never” (#15858)
Fixes #15849
2021-03-12 05:25:50 +01:00
Claire e78f476c58 Remove subscription_expires_at leftover from OStatus (#15857) 2021-03-12 05:25:24 +01:00
Eugen Rochko 4930e71ae7 Add server rules (#15769) 2021-02-21 19:50:12 +01:00
Claire 4028717020 Create instance actor if it hasn't been properly seeded (#15693)
An uncommon but somewhat difficult to digagnose issue is dealing with
improperly-seeded databases. In such cases, instance-signed fetches will
fail with a ActiveRecord::RecordNotFound error, usually caught and handled
as generic 404, leading people to think the remote resource itself has not
been found, while it's the local instance actor that does not exist.

This commit changes the code so that failure to find the instance actor
automatically creates a new one, so that improperly-seeded databases do
not cause any issue.
2021-02-09 18:12:54 +01:00
Eugen Rochko 826a5cc624 Change max. image dimensions to 1920x1080px (1080p) (#15690)
* Change max. image size to 1920x1080px

* Change it in web UI too
2021-02-09 01:21:06 +01:00
ThibG dd2c80e1d1 Fix race conditions on account migration creation (#15597)
* Atomically check for processing lock in Move handler

* Prevent race condition when creating account migrations

Fixes #15595

* Add tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-02-02 14:49:57 +01:00
abcang 2b9d71cc18 Improved performance of notification preloading (#15640)
* Improved performance of notification preloading

* Remove Cacheable from Notification

* Fix test
2021-01-31 21:24:57 +01:00
abcang 82eb3a575e Fix N+1 query when rendering with StatusSerializer (#15641) 2021-01-31 21:24:17 +01:00
luigi c19fe86d66 Optimize some regex matching (#15528)
* Use Regex#match?

* Replace =~ too

* Avoid to call match? from Nil

* Keep value of Regexp.last_match
2021-01-22 10:09:08 +01:00
ThibG 0306d0d4b0 Fix sign-up restrictions based on IP addresses not being enforced (#15607)
Fixes #15606

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-21 06:18:30 +01:00
ThibG 2582047c36 Use Rails' index_by where it makes sense (#15542)
* Use Rails' index_by where it makes sense

* Fix tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-12 09:27:38 +01:00
luigi 944b059f50 Optimize map { ... }.compact calls (#15513)
* Optimize map { ... }.compact

using Enumerable#filter_map, supported since Ruby 2.7

* Add poyfill for Enumerable#filter_map
2021-01-10 00:32:01 +01:00
abcang 41f9aa5dbb Fix rubocop config and warnings (#15503)
* disable NewCops

* update TargetRubyVersion

* Fix Lint/MissingSuper for ActiveModelSerializers::Model

* Fix Lint/MissingSuper for feed

* Fix Lint/FloatComparison

* Do not use instance variables
2021-01-07 09:40:55 +01:00
ThibG 82d014fd08 Fix being able to import more than allowed number of follows (#15384)
* Fix being able to import more than allowed number of follows

Without this commit, if someone tries importing a second list of accounts to
follow before the first one has been processed, this will queue imports for
the two whole lists, even if they exceed the account's allowed number of
outgoing follows.

This commit changes it so the individual queued imports aren't exempt from
the follow limit check (they remain exempt from the rate-limiting check
though).

* Catch validation errors to not re-queue failed follows

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-26 23:52:46 +01:00
ThibG 8d75aa630f Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
ThibG f894bb9220 Fix external user creation failing when invite request text is required (#15405)
* Fix external user creation failing when invite request text is required

Also fixes tootctl-based user creation.

* Add test about invites when invite request text is otherwise required

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 17:14:32 +01:00
Eugen Rochko 3489758ff2 Fix unnecessary queries when batch-removing statuses, 100x faster (#15387) 2020-12-22 17:13:55 +01:00
ThibG 806f7ea54f Fix some notifications not being deleted on poll/status deletion (#15402)
* Fix deleting polls not deleting notifications

* Fix fav notification deletion when deleting a toot

* Refactor DeleteAccountService spec

* Add DeleteAccountService tests for other associations and notifications

* Add favourite handling spec in status removal

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-21 18:22:17 +01:00
ThibG ad1a3c6435 Fix invitation links not working when invite request text is required (#15385)
Fixes #15383

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-21 00:47:02 +01:00
ThibG bbf0e7107c Fix AccountDeletionWorker crashing and clogging sidekiq queues (#15380)
* Fix account deletion workers being queued multiple times for a single account

* Fix poll votes being unnecessarily instantiated on poll deletion

* Fix favourites being unnecessarily instantiated on status deletion

* Remove inaccurate comments

* Delete polls instead of destroying them

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-20 18:25:00 +01:00
Eugen Rochko e66964d8f0 Fix follow limit preventing re-following of a moved account (#14207) 2020-12-18 09:18:31 +01:00
Eugen Rochko 816df80b86 Add option to obfuscate domain name in public list of domain blocks (#15355)
- Replace the middle of the domain with * characters (except for periods)
- Add SHA-256 digest of the domain name in tooltip
2020-12-18 08:30:41 +01:00
ThibG 02bd1060d8 Fix admins being able to suspend their instance actor (#14567)
* Fix admin being able to suspend their own instance account

* Add text about the instance's own actor in admin view

* Change instance actor notice from flash message to template

* Do not list local instance actor in account moderation list
2020-12-15 17:23:58 +01:00
ThibG 3debd888a6 Add indication to admin UI of whether a report has been forwarded (#13237)
* Add indication to admin UI of whether a report has been forwarded

* Rework how forwarded status is displayed

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-15 04:30:15 +01:00
ThibG 64eaaff345 Add ability to require invite request text (#15326)
Fixes #15273

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-14 10:03:09 +01:00
Eugen Rochko df8874b24e Fix performance on instances list in admin UI (#15282)
- Reduce duplicate queries
- Remove n+1 queries
- Add accounts count to detailed view
- Add separate action log entry for updating existing domain blocks
2020-12-14 09:06:34 +01:00
ThibG e1ef5f3b31 Add honeypot fields and minimum fill-out time for sign-up form (#15276)
* Add honeypot fields to limit non-specialized spam

Add two honeypot fields: a fake website input and a fake password confirmation
one. The label/placeholder/aria-label tells not to fill them, and they are
hidden in CSS, so legitimate users should not fall into these.

This should cut down on some non-Mastodon-specific spambots.

* Require a 3 seconds delay before submitting the registration form

* Fix tests

* Move registration form time check to model validation

* Give people a chance to clear the honeypot fields

* Refactor honeypot translation strings

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-10 06:27:26 +01:00
trwnh eb9be7be0c rename replies_policy enumerables (#15304) 2020-12-09 04:34:17 +01:00
Eugen Rochko 48534762cf Fix slow distinct queries where grouped queries are faster (#15287)
About 2x speed-up on inboxes query
2020-12-07 12:08:30 +01:00
ThibG 1fce69291f Fix account merging in maintenance script (#15264)
Also include AccountNote and AccountDeletionRequest to the list of classes
needing to be reassigned the merged account.
2020-12-04 02:23:51 +01:00
ThibG 39824d29f1 Fix followers hash cache not being invalidated on account merge (#15256)
Also clear relationships cache.
2020-12-02 21:20:00 +01:00
Eugen Rochko 876df5bb18 Add `tootctl accounts merge` (#15201)
* Add `tootctl accounts merge`

* Update lib/mastodon/accounts_cli.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2020-11-23 17:50:16 +01:00
ThibG 8e79bac43d Add import/export feature for bookmarks (#14956)
* Add ability to export bookmarks

* Add support for importing bookmarks

* Add bookmark import tests

* Add bookmarks export test
2020-11-19 17:48:13 +01:00
Eugen Rochko 7e2920d9b2 Fix streaming API allowing connections to persist after access token invalidation (#15111)
Fix #14816
2020-11-12 23:05:24 +01:00
Eugen Rochko 90faa8039c Fix 2FA/sign-in token sessions being valid after password change (#14802)
If someone tries logging in to an account and is prompted for a 2FA
code or sign-in token, even if the account's password or e-mail is
updated in the meantime, the session will show the prompt and allow
the login process to complete with a valid 2FA code or sign-in token
2020-11-12 23:05:01 +01:00
ThibG 472b0e77e7 Fix possible inconsistencies in tag search (#14906)
Do not downcase the queried tag before passing it to postgres when searching:
- tags are not downcased on creation
- `arel_table[:name].lower.matches(pattern)` generates an ILIKE anyway
- if Postgres and Rails happen to use different case-folding rules,
  downcasing before query but not before insertion may mean that some
  tags with some casings are not searchable
2020-11-12 18:35:23 +01:00
Takeshi Umeda 1dde6183a3 Add follow selected followers button (#15148)
* Add follow selected followers button

* Fix unused variable

* Fix i18n normalize
2020-11-12 16:58:00 +01:00
Eugen Rochko f9dfbaf6b2 Fix updating account counters when account_stat is not yet created (#15108) 2020-11-09 16:00:23 +01:00
Eugen Rochko 38be9af436 Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
ThibG b1a78a5cce Change order of announcements in admin page to sort them newest-first (#15091)
* Change order of announcements in admin page to sort show newly-created first

Fixes #15090

* Use reverse-chronological rather than creation date only
2020-11-04 21:15:22 +01:00
Takeshi Umeda 385c2eac6f Add account sensitized (#14361)
* Add account sensitized

* Fix i18n normalize

* Fix description and spec

* Fix spec

* Fix wording
2020-11-04 20:45:01 +01:00
ThibG 694a5e33d5 Add follower synchronization mechanism (#14510)
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
OSAMU SATO ba9252b32d Add duration parameter to muting. (#13831)
* Adding duration to muting.

* Remove useless checks
2020-10-13 01:01:14 +02:00
Eugen Rochko da4c730c47 Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
Eugen Rochko adaf3f7b7a Remove dependency on goldfinger gem (#14919)
There are edge cases where requests to certain hosts timeout when
using the vanilla HTTP.rb gem, which the goldfinger gem uses. Now
that we no longer need to support OStatus servers, webfinger logic
is so simple that there is no point encapsulating it in a gem, so
we can just use our own Request class. With that, we benefit from
more robust timeout code and IPv4/IPv6 resolution.

Fix #14091
2020-10-08 00:34:57 +02:00
ThibG 79b2d30a19 Add option to disable swiping motions across the WebUI (#13885)
Fixes #13882
2020-09-30 19:31:03 +02:00
Takeshi Umeda 67ee32365d Fix a slow query for TagFeed (#14861)
* Fix a slow query for TagFeed

* rename tags to tag_ids
2020-09-23 16:01:54 +02:00
Eugen Rochko 02c4c91faa Add option to be notified when a followed user posts (#13546)
* Add bell button

Fix #4890

* Remove duplicate type from post-deployment migration

* Fix legacy class type mappings

* Improve query performance with better index

* Fix validation

* Remove redundant index from notifications
2020-09-18 17:26:45 +02:00
kawaguchi f13d563cb7 Fix validates :sign_count of WebauthnCredential (#14806) 2020-09-16 20:16:46 +02:00
Eugen Rochko e514304a76 Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
Eugen Rochko face3be227 Fix reported statuses not being included in warning e-mail (#14778) 2020-09-13 12:52:17 +02:00
tateisu 9d4769941f allow pagination by min_id and max_id (#14776)
* allow pagination by min_id and max_id

* also AccountConversation allows min_id,max_id pair

* also home,list TL allows min_id,max_id pair
2020-09-12 17:09:49 +02:00
Eugen Rochko 7fa0da9e98 Change REST API to return empty data for suspended accounts (#14765) 2020-09-11 15:16:29 +02:00
abcang e8c39853ad Changed tag most_used to recently_used (#14760) 2020-09-07 17:47:41 +02:00
Eugen Rochko 78ceda1af6 Refactor how public and tag timelines are queried (#14728) 2020-09-07 11:02:04 +02:00
ThibG 50d0c1e95f Add configuration option to filter replies in lists (#9205)
* Add database support for list show-reply preferences

* Add backend support to read and update list-specific show_replies settings

* Add basic UI to set list replies setting

* Add specs for list replies policy

* Switch "cycling" reply policy link to a set of radio inputs

* Capitalize replies_policy strings

* Change radio button design to be consistent with that of the directory explorer
2020-09-01 13:31:28 +02:00
dependabot[bot] 61b768572e Bump rubocop from 0.86.0 to 0.88.0 (#14412)
* Bump rubocop from 0.86.0 to 0.88.0

Bumps [rubocop](https://github.com/rubocop-hq/rubocop) from 0.86.0 to 0.88.0.
- [Release notes](https://github.com/rubocop-hq/rubocop/releases)
- [Changelog](https://github.com/rubocop-hq/rubocop/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rubocop-hq/rubocop/compare/v0.86.0...v0.88.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Fix for latest RuboCop

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2020-09-01 03:04:00 +02:00
Akihiko Odaki b096ad1c0f Make Array-creation behavior of Paginable more predictable (#14687)
* Make Array-creation behavior of Paginable more predictable

Paginable.paginate_by_id usually returns ActiveRecord::Relation, but it
returns an Array if min_id option is present. The behavior caused problems
fixed with the following commits:
- 976938bf30
- 5a8fa3ad95
- 1c70d8189c

To prevent from recurring similar problems, this commit introduces two
changes:
- The scope now always returns an Array whether min_id option is present
  or not.
- The scope is renamed to to_a_paginated_by_id to clarify it returns an
  Array.

* Transform Paginable.to_a_paginated_by_id from a scope to a class method

https://api.rubyonrails.org/classes/ActiveRecord/Scoping/Named/ClassMethods.html#method-i-scope
> The method is intended to return an ActiveRecord::Relation object, which
> is composable with other scopes.

Paginable.to_a_paginated_by_id returns an Array and is not appropriate
as a scope.
2020-08-31 12:47:09 +02:00
Eugen Rochko 97762d28b6 Fix videos with near-60 fps being rejected (#14684)
Fix #14668
2020-08-30 01:54:30 +02:00
santiagorodriguez96 f142983484 Add WebAuthn as an alternative 2FA method (#14466)
* feat: add possibility of adding WebAuthn security keys to use as 2FA

This adds a basic UI for enabling WebAuthn 2FA. We did a little refactor
to the Settings page for editing the 2FA methods – now it will list the
methods that are available to the user (TOTP and WebAuthn) and from
there they'll be able to add or remove any of them.
Also, it's worth mentioning that for enabling WebAuthn it's required to
have TOTP enabled, so the first time that you go to the 2FA Settings
page, you'll be asked to set it up.
This work was inspired by the one donde by Github in their platform, and
despite it could be approached in different ways, we decided to go with
this one given that we feel that this gives a great UX.

Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>

* feat: add request for WebAuthn as second factor at login if enabled

This commits adds the feature for using WebAuthn as a second factor for
login when enabled.
If users have WebAuthn enabled, now a page requesting for the use of a
WebAuthn credential for log in will appear, although a link redirecting
to the old page for logging in using a two-factor code will also be
present.

Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>

* feat: add possibility of deleting WebAuthn Credentials

Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>

* feat: disable WebAuthn when an Admin disables 2FA for a user

Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>

* feat: remove ability to disable TOTP leaving only WebAuthn as 2FA

Following examples form other platforms like Github, we decided to make
Webauthn 2FA secondary to 2FA with TOTP, so that we removed the
possibility of removing TOTP authentication only, leaving users with
just WEbAuthn as 2FA. Instead, users will have to click on 'Disable 2FA'
in order to remove second factor auth.
The reason for WebAuthn being secondary to TOPT is that in that way,
users will still be able to log in using their code from their phone's
application if they don't have their security keys with them – or maybe
even lost them.

* We had to change a little the flow for setting up TOTP, given that now
  it's possible to setting up again if you already had TOTP, in order to
  let users modify their authenticator app – given that now it's not
  possible for them to disable TOTP and set it up again with another
  authenticator app.
  So, basically, now instead of storing the new `otp_secret` in the
  user, we store it in the session until the process of set up is
  finished.
  This was because, as it was before, when users clicked on 'Edit' in
  the new two-factor methods lists page, but then went back without
  finishing the flow, their `otp_secret` had been changed therefore
  invalidating their previous authenticator app, making them unable to
  log in again using TOTP.

Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>

* refactor: fix eslint errors

The PR build was failing given that linting returning some errors.
This commit attempts to fix them.

* refactor: normalize i18n translations

The build was failing given that i18n translations files were not
normalized.
This commits fixes that.

* refactor: avoid having the webauthn gem locked to a specific version

* refactor: use symbols for routes without '/'

* refactor: avoid sending webauthn disabled email when 2FA is disabled

When an admins disable 2FA for users, we were sending two mails
to them, one notifying that 2FA was disabled and the other to notify
that WebAuthn was disabled.
As the second one is redundant since the first email includes it, we can
remove it and send just one email to users.

* refactor: avoid creating new env variable for webauthn_origin config

* refactor: improve flash error messages for webauthn pages

Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>
2020-08-24 16:46:27 +02:00
niwatori24 65c9d28fa1 Cache result of SQL (#14534) 2020-08-09 15:23:42 +02:00
Takeshi Umeda 622dcbeb8c Fix movie width and frame_rate returning nil (#14357)
* Fix movie width and frame_rate returning nil

* Add StreamValidationError and raise

* Fix code style
2020-07-19 22:28:27 +02:00
abcang aca93fa882 Fix rubocop warning (#14288)
* Fix rubocop warning

* use limit variable

* use ContextCreatingMethods option
2020-07-14 19:05:07 +02:00
ThibG d93a24936c Fix larger video files not being transcoded (#14306)
Since #14145, the `set_type_and_extension` has been moved from
`before_post_process` to `before_file_post_process`, but while the former
runs before all validations performed by Paperclip, the latter is dependent
on the order validations and hooks are defined.

In our case, this meant video files could be checked against the generic 10MB
limit, causing validation failures, which, internally, make Paperclip skip
post-processing, and thus, transcoding of the video file.

The actual validation would then happen after the type is correctly set, so
the large file would pass validation, but without being transcoded first.

This commit moves the hook definition so that it is run before checking for
the file size.
2020-07-14 18:50:19 +02:00
Eugen Rochko 8517a5fdb4 Add color extraction for audio thumbnails (#14209) 2020-07-05 18:28:25 +02:00
Eugen Rochko ca238c879d Change Redis#exists calls to Redis#exists? to avoid deprecation warning (#14191) 2020-07-01 19:05:21 +02:00
Eugen Rochko 36e5fd547b Fix remote files not using Content-Type header, streaming (#14184) 2020-06-30 23:58:02 +02:00
ThibG 3afc7977b1 Add user notes on accounts (#14148)
* Add UserNote model

* Add UI for user notes

* Put comment in relationships entity

* Add API to create user notes

* Copy user notes to new account when receiving a Move activity

* Address some of the review remarks

* Replace modal by inline edition

* Please CodeClimate

* Button design changes

* Change design again

* Cancel note edition when pressing Escape

* Fixes

* Tweak design again

* Move “Add note” item, and allow users to add notes to themselves

* Rename UserNote into AccountNote, rename “comment” Relationship attribute to “note”
2020-06-30 19:19:50 +02:00
Eugen Rochko 16a8db0958 Fix trying to write non-existent image remote URL attribute on preview cards (#14181)
Regression from #14145
2020-06-29 17:59:04 +02:00
Eugen Rochko a3ce01a102 Add customizable thumbnails for audio and video attachments (#14145)
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
2020-06-29 13:56:55 +02:00
Eugen Rochko 12ac904a33 Fix various issues around OpenGraph representation of media (#14133)
- Fix audio attachments not being represented in OpenGraph tags
- Fix audio being represented as "1 image" in OpenGraph descriptions
- Fix video metadata being overwritten by paperclip-av-transcoder
- Fix embedded player not using Mastodon's UI
- Fix audio/video progress bars not moving smoothly
- Fix audio/video buffered bars not displaying correctly
2020-06-25 01:33:01 +02:00
Eugen Rochko 2dbf6bc5ad Add e-mail-based sign in challenge for users with disabled 2FA (#14013) 2020-06-09 10:23:06 +02:00
ThibG 36f31b5cad Add blurhash to preview cards (#13984)
Fixes #13001
2020-06-05 23:10:41 +02:00
Eugen Rochko b864e78db8 Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
ThibG 4255757678 Fix admin page crashing when trying to block an invalid domain name (#13884)
* Fix admin page crashing when trying to block an invalid domain name

Fixes #13880

* Fix trailing and leading spaces not being properly stripped for domain blocks
2020-06-01 03:47:20 +02:00
Takeshi Umeda e2694a18c7 Fix csv upload (#13835) 2020-05-24 09:15:23 +02:00
Takeshi Umeda 56abcac438 Fix searchable status without explicitly mentioning it (#13829) 2020-05-23 05:47:25 +02:00
dependabot-preview[bot] e7e09d3b7b Bump doorkeeper from 5.3.3 to 5.4.0 (#13733)
* Bump doorkeeper from 5.3.3 to 5.4.0

Bumps [doorkeeper](https://github.com/doorkeeper-gem/doorkeeper) from 5.3.3 to 5.4.0.
- [Release notes](https://github.com/doorkeeper-gem/doorkeeper/releases)
- [Changelog](https://github.com/doorkeeper-gem/doorkeeper/blob/master/CHANGELOG.md)
- [Commits](https://github.com/doorkeeper-gem/doorkeeper/compare/v5.3.3...v5.4.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Fix tests

* Fix use of Doorkeeper::AccessToken.find_or_create_for

* Fix tests?

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Thibaut Girka <thib@sitedethib.com>
2020-05-12 15:25:33 +02:00
Eugen Rochko 3144f8dbbc Refactor monkey-patching of Goldfinger (#12561) 2020-05-10 11:41:43 +02:00
Takeshi Umeda 861661100c Add remote only to public timeline (#13504)
* Add remote only to public timeline

* Fix code style
2020-05-10 10:36:18 +02:00
ThibG 41536557be Improve RSS entries for statuses (#13592)
* Improve RSS entries for statuses

- Render polls in both accounts and tags serializers
- Refactor RSS serializers
- Change title preview to include ellipsis when truncated
- Change title preview to show CW instead of toot text
- Add tests

* Remove title from OEmbed serialization

Twitter doesn't serialize title either, and tihs allows us to move the
title formatting code to the RSS serializers.
2020-05-10 09:50:54 +02:00
Taras Gogol 1a03de8592 Fix followings list order | Issue #13538 (#13676) 2020-05-08 20:17:16 +02:00
Yamagishi Kazutoshi cf55546c41 Revert "improve status title (#8596)" (#13591)
This reverts commit dff05cd398.
2020-05-03 18:48:13 +02:00
Eugen Rochko 0ca06d0ba9 Add more tests for ActivityPub controllers (#13585) 2020-05-03 16:30:36 +02:00
kaiyou 7a65e2f0db Only check locally when deduplicating usernames (#13581)
When deduplicating account usernames for OAuthable users, the routine did check if any account was known with that username, including remote accounts. This caused some unnecessary deduplication, and usernames ending with unexpected trailing _1.
This fixes #13580
2020-04-30 14:39:05 +02:00
Eugen Rochko adfb22a9dd Add separate cache directory for non-local uploads (#12821) 2020-04-26 23:29:08 +02:00
Takeshi Umeda f365b9d6cd Fix enable/disable relay failures (#13535) 2020-04-23 22:04:18 +02:00
ThibG c162748a14 Fix confusing error when failing to add an alias to an unknown account (#13480)
Follow-up to #13452, fixing broken `uri.nil?` test.

Also remove the separate check for `uri` presence, as that would result
in a “Please review 2 errors below” while only one would be listed.
2020-04-15 20:33:53 +02:00
Eugen Rochko 2d3219549b Change delivery failure tracking to work with hostnames instead of URLs (#13437) 2020-04-15 20:33:24 +02:00
ThibG add34dd132 Fix “Email changed” notification sometimes having wrong e-mail (#13475)
* Fix “Email changed” notification sometimes having wrong e-mail

Fixes #6778

The root of the issue is that `send_devise_notification` was called before
the changes were properly commited to the database, causing the mailer to
pick previous values if running too early.

Devise's documentation provides guidance on how to handle that[1][2], however,
I have found it to not be working, as the following happens, in that order:
- `send_devise_notification` is called for the `email_changed` notification.
  In that case, `changed?` is false and `saved_changes?` is true, so
  if we use the former, we have the same issue.
- the `after_commit` hook is called
- `send_devise_notification` is called for the `confirmation_instructions`
  notification.
  In that case, `changed?` is still false, and `saved_changes?` still true,
  so if we use the latter, that second notification email is simply not
  going to be sent (as we would be queuing the notification *after*
  executing the after_commit hook).

This is because it may be called from either an `after_update` or
`after_commit` hook, the difference not being a call to `save` but the
transaction actually being committed to the database. This may arguably
be a bug in Devise, or Devise's notification.

The proposed workaround is inspired by Devise's documentation but checks
whether a transaction is open to make the call whether to immediately
send the notification or defer it to the `after_commit` hook.

[1]: https://www.rubydoc.info/github/plataformatec/devise/Devise%2FModels%2FAuthenticatable:send_devise_notification
[2]: 406915cb78/lib/devise/models/authenticatable.rb (L133-L194)

* Fix cases when sending notifications without changing the model

* Defer sending if and only if in transaction including current record
2020-04-15 16:13:44 +02:00
ThibG dedac235bc Fix account aliases page (#13452)
* Fix error not being displayed when adding an account alias, add error for self-references

Co-Authored-By: Mélanie Chauvel (ariasuni) <perso@hack-libre.org>

* Add “You have no aliases.” note in confusing empty aliases table

Co-Authored-By: Mélanie Chauvel (ariasuni) <perso@hack-libre.org>

Co-authored-by: Mélanie Chauvel (ariasuni) <perso@hack-libre.org>
2020-04-13 06:41:43 +02:00
Eugen Rochko 50fa554899 Add rate limit for reporting (#13390) 2020-04-05 14:40:08 +02:00
ThibG ef48ffc127 Fix PostgreSQL load when linking in announcements (#13250)
* Fix PostgreSQL load when linking in announcements

Fixes #13245 by caching status lookups

Since statuses are supposed to be known already and we only
need their URLs and a few other things, caching them should
be fine.

Since it's only used by announcements so far, there won't
be much statuses to cache.

* Perform status lookup when saving announcements, not when rendering them

* Change EntityCache#status to fetch URLs instead of looking into the database

* Move announcement link lookup to publishing worker

* Address issues pointed out during review
2020-04-05 12:51:22 +02:00
Eugen Rochko 6932e0e2af Add ability to filter audit log in admin UI (#13381) 2020-04-03 13:06:34 +02:00
Eugen Rochko 1cb4645638 Change `tootctl media remove-orphans` to work for all classes (#13316)
Change `tootctl media lookup` to not use an interactive prompt
2020-03-26 01:56:41 +01:00
Jeong Arm 3b4e54f359 Make bookmarks also searchable (#13271) 2020-03-21 03:14:10 +01:00
ThibG 977b7ef6d0 Fix reported accounts not being whitelisted when resolving a spamcheck report (#13289) 2020-03-21 03:08:09 +01:00
Eugen Rochko 9ced8b6dd5 Add option to include resolved DNS records when blacklisting e-mail domains in admin UI (#13254)
* Add shortcuts to blacklist a user's e-mail domain in admin UI

* Add option to blacklist resolved MX and IP records for e-mail domains
2020-03-12 22:35:20 +01:00
Eugen Rochko da9d81c4ac Add titles to warning presets in admin UI (#13252) 2020-03-12 17:57:59 +01:00
ThibG 0701e09c85 Fix videos with unsupported colorspace not being transcoded (#13242) 2020-03-10 11:58:40 +01:00
ThibG ccd8ceac47 Fix MP4 (H264 + AAC) video files being needlessly re-encoded (#13239) 2020-03-09 23:15:59 +01:00
Eugen Rochko a0c60d09ff Change video uploads to always be converted to H264/MP4 (#13220)
Even if the container format is the same (.mp4), the codec could
be different and not playable in web browsers
2020-03-09 02:20:18 +01:00
Eugen Rochko 0995464585 Change video uploads to enforce certain limits (#13218)
- Dimensions at most 1920x1200
- Frame rate at most 60
2020-03-09 02:19:07 +01:00
ThibG cd7c22da32 Add federation support for the "hide network" preference (#11673)
* Change ActivityPub follower/following collections to not link first page

* Add support for hiding followers and following of remote users

* Switch to using a single `hide_collections` column

* Address code style remarks
2020-03-09 00:10:29 +01:00
Eugen Rochko f459919552 Change local media attachments to perform heavy processing asynchronously (#13210)
Fix #9106
2020-03-08 23:56:18 +01:00
ThibG a67fd2fbe1 Add support for links to statuses in announcements to be opened in web UI (#13212)
* Add support for links to public statuses in announcements to be opened in WebUI

* Please CodeClimate
2020-03-08 16:10:48 +01:00
ThibG 49f5db61bc Add sorting by username, creation and last activity in moderation view (#13076)
* Add ability to order accounts in moderation view

* Display last status date in “Most recent activity” for remote users
2020-03-08 15:39:13 +01:00
Eugen Rochko a4668d994b Add specific rate limits for posting and following (#13172) 2020-03-08 15:17:39 +01:00
Eugen Rochko df086209a2 Fix "tootctl media remove-orphans" crashing when encountering invalid media (#13170)
Fixes #13168
2020-03-05 15:56:01 +01:00
ThibG 69aefeb7ee Fix announcements with fully-qualified mention to local user crashing WebUI (#13164) 2020-02-28 13:49:45 +01:00
Eugen Rochko d0fe0d348b Fix remote convertible media attachments not being saved (#13032)
Because the file name was set after loading the file, Paperclip
was flagging the differing file extension as content type spoofing

Fix #12938
2020-02-03 18:45:56 +01:00
abcang 94bc823eb6 Search account domain in lowercase (#13016)
* Search account domain in lowercase

* fix rubocop error

* fix spec/models/account_spec.rb
2020-02-01 15:42:24 +01:00
Eugen Rochko 9fb8165712 Add publish/unpublish controls to announcements in admin UI (#12967) 2020-01-27 11:05:33 +01:00
Eugen Rochko 9d3bba9576 Fix expired announcements being re-published (#12964) 2020-01-26 22:43:18 +01:00
Eugen Rochko dd68e44cfa Add streaming API updates for announcements being modified or deleted (#12963)
Change `all_day` to be a visual client-side cue only

Publish immediately if `scheduled_at` is in the past

Add `published_at` and `updated_at` to announcements JSON
2020-01-26 20:07:26 +01:00
Eugen Rochko fb1df3d5d5 Fix user disabling changing activity timestamps, fix nil error (#12943) 2020-01-25 05:22:35 +01:00
Eugen Rochko e4aa4a1c28 Fix password change/reset not immediately invalidating other sessions (#12928)
While making browser requests in the other sessions after a password
change or reset does not allow you to be logged in and correctly
invalidates the session making the request, sessions have API tokens
associated with them, which can still be used until that session
is invalidated.

This is a security issue for accounts that were already compromised
some other way because it makes it harder to throw out the hijacker.
2020-01-24 00:20:38 +01:00
Eugen Rochko 36b1dd934d Fix relationships page not showing results in admin UI (#12934)
Follow-up to #12927
2020-01-24 00:20:23 +01:00
Eugen Rochko f95fd62bac Add announcements (#12662)
* Add announcements

Fix #11006

* Add reactions to announcements

* Add admin UI for announcements

* Add unit tests

* Fix issues

- Add `with_dismissed` param to announcements API
- Fix end date not being formatted when time range is given
- Fix announcement delete causing reactions to send streaming updates
- Fix announcements container growing too wide and mascot too small
- Fix `all_day` being settable when no time range is given
- Change text "Update" to "Announcement"

* Fix scheduler unpublishing announcements before they are due

* Fix filter params not being passed to announcements filter
2020-01-23 22:00:13 +01:00
Eugen Rochko 930ba0630f Fix media attachments without file being uploadable (#12562)
Fix #12554
2020-01-23 21:40:03 +01:00
ThibG 691b14aaf8 Add “account timeline” filter category (#12918)
* Add “account timeline” filter category

Previously, no filter category applied to account timelines.

* Rename “Account timelines” into “Profiles”
2020-01-23 21:32:00 +01:00
Eugen Rochko 67172aa4f9 Change followers page to relationships page in admin UI (#12927)
Allow browsing and filtering all relationships instead of just
followers, unify the codebase with the user-facing relationship
manager, add ability to see who the user invited
2020-01-23 20:33:20 +01:00
Eugen Rochko 9a9eb0f3d4 Fix not all of account's active IPs showing up in admin UI (#12909) 2020-01-20 18:00:54 +01:00
Eugen Rochko ba3d2bd01d Fix search by IP not using alternative browser sessions in admin UI (#12904) 2020-01-20 17:00:41 +01:00
Eugen Rochko 9ebfb23e87 Various fixes and improvements (#12878)
* Fix unused role routes being generated

* Remove unused JavaScript code

* Refactor filters code to be DRYer

* Fix `.count == 0` comparisons to `.empty?` in views

* Fix filters in views
2020-01-20 15:55:03 +01:00
Eugen Rochko ddb8bd13ea Fix file names being obfuscated on update when file didn't change (#12857)
Fix #12849
2020-01-13 12:54:33 +01:00
ThibG 96b43ad187 Remove dependency on OStatus2 gem (#12822) 2020-01-11 21:36:53 +01:00
ysksn 312c00ca35 Refactor StatusThreadingConcern (#9626)
* Remove #filter_from_context?

* Create scope Status.with_accounts

Retrieving AR objects should be
their model's scope
2020-01-11 19:55:33 +09:00
Alexander dff05cd398 improve status title (#8596)
* improve shown status title, useful for atom/rss

* use single quotes to satisfy codeclimate

* fix tests, make message more pretty

* fix tests

* fix codestyle

* fix codestyle

* remove atom_serializer_spec

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2020-01-11 06:58:16 +09:00
ThibG f52ef9b007 Allow blocking TLDs, and fix TLD blocks not being editable (#12805)
Fixes #12795

It was already possible to create domain blocks for TLDs, but those
weren't enforced, nor editable. This commit changes it so that they
are enforced and editable.
2020-01-08 22:42:05 +01:00
Eugen Rochko 413cb7e861 Fix base64-encoded file uploads not being possible (#12748)
Fix #3804, Fix #5776
2020-01-04 01:54:07 +01:00
Eugen Rochko c72c27756f Fix IDN mentions not being processed, IDN domains not being rendered (#12715)
This changes the REST API to return unicode domains in the `acct`
attribute instead of punycode, and to render unicode instead of
punycode on public HTML pages as well.

Fix #7812, fix #12246
2019-12-30 19:20:43 +01:00
ThibG ec3758bf82 Fix custom emoji category creation silently erroring out on duplicate category (#12647)
Instead, just re-use the existing category if any.

Fixes #12608
2019-12-18 16:56:59 +01:00
ThibG f5ac6f7801 Fixes featured hashtag setting page erroring out instead of rejecting invalid tags (#12436)
* Revert "Fix ignoring whole status because of one invalid hashtag (#11621)"

This reverts commit 123ffd0a45.

* Fix statuses being rejected because of invalid hashtag names

* Add spec for invalid hashtag names in statuses

* Add test for featured tags controller
2019-12-17 13:31:56 +01:00
scd31 27f1d5ee7d Increase max backup size (#12602)
* Increased max backup size

* partially reverted schema.rb
2019-12-17 13:30:58 +01:00
Alice Gaudon a285b3c63c Admin setting to disable default follows (#12566) 2019-12-16 23:55:50 +01:00
Eugen Rochko 0769ef8277 Add `tootctl media remove-orphans` (#12568) 2019-12-08 15:37:12 +01:00
Takeshi Umeda 58dd9698c3 Add basic support for group actors (#12071)
* Show badge on group actor in WebUI

* Do not notify in case of  by following group actor

* If you mention group actor, also mention group actor followers

* Relax characters that can be used in username (same as Application)

* Revert "Relax characters that can be used in username (same as Application)"

This reverts commit 7e10a137b878d0db1b5252c52106faef5e09ca4b.

* Delete display_name method
2019-12-04 20:36:33 +01:00
Yamagishi Kazutoshi b01cb4701e Update ESLint and RuboCop in Code Climate (#12534) 2019-12-02 18:25:43 +01:00
Mathieu Brunot 128a7ab7eb Add an LDAP Mail attribute config (#12053)
Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>
2019-12-01 18:52:21 +01:00
ThibG 60d26cabf0 Add follow_request notification type (#12198)
* Add follow_request notification type

The notification type already existed in the backend but was never pushed
to the front-end. This also means translation strings were also available
for the backend, from the notification mailer.

Unlike other notification types, these are off by default, to match what
I remember of Gargron's view on the topic: that follow requests should not
clutter notifications and should instead be reviewed at the user's own
leisure in the dedicated column.

Since follow requests have their own column, I've deemed it unnecessary to
add a specific tab for them in the notification quick filter.

* Show follow request link in single-column if there are pending requests, even if account isn't locked

* Push follow requests from notifications to the follow_requests list

* Offer to accept or reject follow request from the notification

* Redesign follow request notification
2019-12-01 17:25:29 +01:00
ThibG acb1927158 Only normalize local polls (#12515)
Before this patch, if remote poll options have leading or trailing spaces,
the information stored locally won't match them, causing federated voting to
fail.
2019-12-01 17:24:33 +01:00
Mathieu Brunot 462788cb12 Convert LDAP username (#12461)
*  Convert LDAP username #12021

Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>

* 🐛 Fix conversion var use

Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>

* 🐛 Fix LDAP uid conversion test

Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>

* 👌 Remove comments with ref to PR

Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>

* 👌 Remove unnecessary paranthesis

Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>

* 🔧 Move space in conversion string

Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>
2019-12-01 07:21:28 +01:00
ThibG 4b0a6d79dd Add ability to filter reports by target account domain (#12154)
* Add ability to filter reports by target account domain

* Reword by_target_domain label
2019-11-30 19:53:58 +01:00
ThibG c6c0b30608 Simplify SQL query when performing account search amongst followings (#12302) 2019-11-21 17:35:04 +01:00
ThibG 517c1cd062 Add bookmarks (#7107)
* Add backend support for bookmarks

Bookmarks behave like favourites, except they aren't shared with other
users and do not have an associated counter.

* Add spec for bookmark endpoints

* Add front-end support for bookmarks

* Introduce OAuth scopes for bookmarks

* Add bookmarks to archive takeout

* Fix migration

* Coding style fixes

* Fix rebase issue

* Update bookmarked_statuses to latest UI changes

* Update bookmark actions to properly reflect status changes in state

* Add bookmarks item to single-column layout

* Make active bookmarks red
2019-11-13 23:02:10 +01:00
ThibG 6b5aebf0ca Add abilityto add oneself to lists (#12271)
* Add ability to add oneself to lists

* Change search results to include oneself when searching through followers

* Mark follow relation as optional in ListAccount
2019-11-04 13:02:01 +01:00
ThibG 479d8ff47a Fix remote media descriptions being cut off at 420 chars (#12262)
* Fix remote media descriptions being cut off at 420 chars

Fixes #12258

* Fix tests
2019-11-04 13:00:16 +01:00
Faye Duxovni 6bca6dbc2d Add setting for whether to crop images in unexpanded toots (#12126) 2019-10-24 22:51:41 +02:00
Hugo Gameiro 9980046428 remove audio metadata (#12171) 2019-10-24 22:47:58 +02:00
Eugen Rochko cfb06d0565 Fix admin setting to auto-approve hashtags not affecting query (#12130)
Follow-up to #12122
2019-10-10 02:22:04 +02:00
Eugen Rochko d99a6145de Remove auto-silence behaviour from spam check (#12117)
Fix #12113
2019-10-09 07:11:23 +02:00
Eugen Rochko 2e07a901c5 Fix attachment not being re-downloaded even if file is not stored (#12125)
Change the behaviour of remotable concern. Previously, it would skip
downloading an attachment if the stored remote URL is identical to
the new one. Now it would not be skipped if the attachment is not
actually currently stored by Paperclip.
2019-10-09 07:10:46 +02:00
Eugen Rochko 001ce9a1a9 Add admin setting to auto-approve hashtags (#12122)
Change inaccurate labels on other admin settings
2019-10-09 00:30:15 +02:00
Eugen Rochko c453888616 Fix performance of home feed regeneration (#12084)
Fetching statuses from all followed accounts at once takes too long
within Postgres. Fetching them one by one and merging in Ruby
could be a lot less resource-intensive

Because the query for dynamically fetching the home timeline is so
heavy, we can no longer offer it when the home timeline is missing
2019-10-06 22:11:17 +02:00
Hugo Gameiro b302921eb1 add loglevel fatal to video and audio styles (#12088) 2019-10-06 19:48:26 +02:00
Eugen Rochko 746fae0bd1 Fix performance of GIF re-encoding (#12057)
* Change animated GIF detection to not shell out to ImageMagick

Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>

* Change video encoding parameters to limit to 10800 video frames

Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>

* Limit GIF image size further

Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>

* Always strip metadata from video files

* Fix code style issues
2019-10-03 01:09:12 +02:00
Eugen Rochko 1db0719dbb Fix needlessly indexing unsearchable statuses into ElasticSearch (#12041) 2019-10-02 20:04:46 +02:00
Eugen Rochko ced431f02f Fix account counters being overwritten by parallel writes (#12045) 2019-10-02 04:59:37 +02:00
Eugen Rochko f77337daaf Fix records not being indexed sometimes (#12024)
It's possible that after commit callbacks were not firing when
exceptions occurred in the process. Also, the default Sidekiq
strategy does not push indexing jobs immediately, which is not
necessary and could be part of the issue too.
2019-10-01 01:19:11 +02:00
ThibG f088e090e4 Add voters count support (#11917)
* Add voters count to polls

* Add ActivityPub serialization and parsing of voters count

* Add support for voters count in WebUI

* Move incrementation of voters count out of redis lock

* Reword “voters” to “people”
2019-09-29 22:58:01 +02:00
Eugen Rochko f5c71a6cd2 Add (back) option to set redirect notice on account without moving followers (#11994)
Fix #11913
2019-09-29 05:03:19 +02:00
Eugen Rochko d8f09d5853 Fix unscoped being used in the wrong place (#11987)
Regression from #11982
2019-09-28 05:23:32 +02:00
Eugen Rochko 386dc65671 Fix preview card image not being re-fetched even if link is re-posted (#11981)
Fix #11956
2019-09-28 01:33:16 +02:00
Eugen Rochko 6baaceefcd Fix index not being used in Status.reblogs_map (#11982)
Regression from #11623
2019-09-28 01:33:02 +02:00
Eugen Rochko 66d3e13777 Add `exclude_unreviewed` param to `GET /api/v2/search` REST API (#11977)
Make it so normal search returns even unreviewed matches, but
autosuggestions do not.

Fix #11960
2019-09-28 01:02:21 +02:00
abcang 2c43d538dc Validate Web::PushSubscription (#11971) 2019-09-27 15:24:13 +02:00
Eugen Rochko 67796a267d Fix relays UI being available in whitelist/secure mode (#11963)
Fix relays UI referencing relay that is not functional
2019-09-27 02:13:34 +02:00
Eugen Rochko 3773115066 Fix authentication before 2FA challenge (#11943)
Regression from #11831
2019-09-24 04:35:36 +02:00
ThibG c596206f8c Show user what options they have voted (#11195)
* Add own_votes field to poll results in REST API

Fixes #10679

* Display user votes in WebUI

* Update styling

* Add vote checkmark to public pages
2019-09-22 14:15:18 +02:00
Takeshi Umeda 9267e4310d Accept acct starting with @ in account migration (#11907) 2019-09-21 09:11:38 +02:00
Takeshi Umeda 45fe523a18 Addition of update activity distribution by alias, minor correction (#11905)
* Addition of update activity distribution by alias, minor correction

* Distribute Update activity after adding alias
* Add uniqueness verification to alias uri
* accept acct starting with @

* fix double-quoted to single-quoted
2019-09-21 09:11:21 +02:00
Eugen Rochko a2d3728c36 Add account migration UI (#11846)
Fix #10736

- Change data export to be available for non-functional accounts
- Change non-functional accounts to include redirecting accounts
2019-09-19 20:58:19 +02:00
Eugen Rochko 61442032a2 Add table of contents to about page (#11885)
Move public domain blocks information to about page
2019-09-19 11:09:05 +02:00
Eugen Rochko 1781358bd9 Add password challenge to 2FA settings, e-mail notifications (#11878)
Fix #3961
2019-09-18 16:37:27 +02:00
Eugen Rochko 8904a4eb36 Fix TOTP codes not being filtered from logs during enabling/disabling (#11877)
Not a serious issue because they are meaningless past single use
2019-09-18 02:48:40 +02:00
han@highemelry 66241cfc7e Fix audit log error when custom emoji is copied from remote server (#11876) 2019-09-17 23:20:48 +02:00
mayaeh 2d79552c3c Fix the sample account icon of Profile directory to local only (#11872) 2019-09-17 10:06:43 +02:00
Eugen Rochko 68ce208da3 Fix SSO login not using existing account when e-mail is verified (#11862)
Fix #11472
2019-09-16 20:42:29 +02:00
mayaeh 566e85716f Add search and sort functions to hashtag admin UI (#11829)
* Add search and sort functions to hashtag admin UI

* Move scope processing from tags_controller to tag_filter

* Fix based on method naming conventions

* Fixed not to get 500 errors for invalid requests
2019-09-16 14:27:29 +02:00
Eugen Rochko 8eb0d880cb Fix 2FA challenge and password challenge for non-database users (#11831)
* Fix 2FA challenge not appearing for non-database users

Fix #11685

* Fix account deletion not working when using external login

Fix #11691
2019-09-15 21:08:39 +02:00
Eugen Rochko 45b759a2ca Fix hashtags being split by ZWNJ character (#11821)
Fix #11761
2019-09-13 16:01:26 +02:00
Eugen Rochko ec20fd2112 Change unlisted custom emoji to not appear in autosuggestions (#11818)
Fix #11669
2019-09-13 16:01:09 +02:00
Eugen Rochko 9301574423 Change max length of media descriptions from 420 to 1500 characters (#11819)
Fix #11658
2019-09-13 16:00:34 +02:00
Eugen Rochko 4291b74031 Change deletes to preserve soft-deleted statuses in unresolved reports (#11805)
Change all account actions except "none" to resolve all unresolved reports

Refactor `SuspendAccountService` to be more readable
2019-09-11 16:32:44 +02:00
Tao Bror Bojlén 0430cbec29 add admin setting for default search engine indexing (fix #11750) (#11804) 2019-09-11 08:44:58 +02:00
Eugen Rochko 8eab4349b4 Add retry for failed media downloads and `tootctl media refresh` (#11775) 2019-09-10 15:29:12 +02:00
Eugen Rochko 666c1a8a06 Change tootctl to use inline parallelization instead of Sidekiq (#11776)
- Remove --background option
- Add --concurrency(=5) option
- Add progress bars
2019-09-10 13:48:48 +02:00
Eugen Rochko 25fb124ee6 Add batch actions and categories to admin UI for custom emojis (#11793) 2019-09-09 22:44:17 +02:00
Eugen Rochko 7799c7c75f Add batch approve/reject for pending hashtags in admin UI (#11791) 2019-09-09 12:50:09 +02:00