Commit Graph

506 Commits (2e46fca321480078d720cadde3aa1d409a17844a)

Author SHA1 Message Date
Eugen Rochko 9a8784829f Fix warnings on Rails boot (#16946) 2021-12-27 00:47:20 +01:00
Claire d8da1689c1 Add ability for admins to delete canonical email blocks (#16644)
* Add admin option to remove canonical email blocks from a deleted account

* Add tootctl canonical_email_blocks to inspect and remove canonical email blocks
2021-12-17 23:02:14 +01:00
Takeshi Umeda 6546346818 Add remove orphans to tootctl statuses remove (#17067)
* Add remove orphans to tootctl statuses remove

* Add REINDEX and change option from vacuum to compression-database

* Changed to extract the deletion target of conversations to a temporary table

* Support progress bar and exceptions when media remove

* Add continue option

* Fix compression to compress

* Remove skip_remove_orphans
2021-12-12 06:09:14 +01:00
Takeshi Umeda 3b1320f817 Fix performance of tootctl statuses remove (#17052)
* Fix performance of tootctl statuses remove

* Fix model class
2021-11-26 22:08:47 +01:00
Takeshi Umeda 8b87566072 Fix ElasticSearch to Elasticsearch (#17050) 2021-11-26 08:30:02 +01:00
OSAMU SATO 20ad635c3c Add batch_size option to bin/tootctl search deploy (#17049) 2021-11-26 08:29:53 +01:00
Eugen Rochko ad73becf3e Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Takeshi Umeda 869faf5f87 Bump chewy from 5.2.0 to 7.2.3 (supports Elasticsearch 7.x) (#16915)
* Bump chewy from 5.2.0 to 7.2.2

* fix style (codeclimate)

* fix style

* fix style

* Bump chewy from 7.2.2 to 7.2.3
2021-11-18 22:02:08 +01:00
Claire ef730593f4 Fix upload of remote media with OpenStack Swift sometimes failing (#16998)
Under certain conditions, files fetched from remotes trigger an error when
being uploaded using OpenStack Swift. This is because in some cases, the
remote server will not return a content-length, so our ResponseWithLimitAdapter
will hold a `nil` value for `#size`, which will lead to an invalid value
for the Content-Length header of the Swift API call.

This commit fixes that by taking the size from the actually-downloaded file
size rather than the upstream-provided Content-Length header value.
2021-11-16 21:36:28 +01:00
Claire 65a727c888 Fix "bundle exec rails mastodon:setup" crashing in some circumstances (#16976)
Fix regression from #16896
2021-11-11 14:00:30 +01:00
Eugen Rochko 8925737d2c Forward port version bumps to 3.4.2 and 3.4.3 (#16945)
* Bump version to 3.4.2

* Bump version to 3.4.3
2021-11-06 05:32:14 +01:00
Jeong Arm a38eceedb1 Skip blocked domains media on tootctl media refresh (#16914) 2021-10-28 19:30:44 +02:00
Claire ee098c8342 Fix mastodon:setup to take dotenv/docker-compose differences into account (#16896)
In order to work around https://github.com/mastodon/mastodon/issues/16895,
add a warning to .env.production.sample, and change the mastodon:setup rake
task to:
- output a warning if a variable will be interpreted differently by dotenv
  and docker-compose
- ensure the printed config is compatible with docker-compose
2021-10-25 16:34:15 +02:00
Claire c1aea260c7 Add S3_FORCE_SINGLE_REQUEST env var to work around S3 compatibility issues (#16866)
Fixes #16822
2021-10-18 18:29:04 +02:00
Jeong Arm 5f554f1171 Add optional domain restrict to tootctl accounts cull (#16511)
* Add optional domain restrict to accounts cull

* Use "unless" - codeclimate
2021-10-14 21:09:56 +02:00
Claire f940e4871d Fix `tootctl accounts cull` not excluding domains on timeouts and certificate issues (#16433)
Fixes #16410
2021-10-14 21:08:37 +02:00
Claire 15d35ed5fa Fix tootctl self-destruct not sending Delete activities for recently-suspended accounts (#16688)
* Do not block existing users' emails on self-destruct

That is wasteful and unintuitive

* Do not close registrations when running tootctl self-destruct with --dry-run

* Close registrations on self-destruct regardless of known remote accounts

* Fix tootctl self-destruct not sending Deletes for recently-suspended accounts

* Suspend local users even if no remote account is known

* Do not show scary confirmation text if ran with --dry-run
2021-10-14 19:59:28 +02:00
Claire 7d0089033f Switch from unmaintained paperclip to kt-paperclip (#16724)
* Switch from unmaintained paperclip to kt-paperclip

* Drop some compatibility monkey-patches not required by kt-paperclip

* Drop media spoof check monkey-patching

It's broken with kt-paperclip and hopefully it won't be needed anymore

* Fix regression introduced by paperclip 6.1.0

* Do not rely on pathname to call FastImage

* Add test for ogg vorbis file with cover art

* Add audio/vorbis to the accepted content-types

This seems erroneous as this would be the content-type for a vorbis stream
without an ogg container, but that's what the `marcel` gem outputs, so…

* Restore missing for_as_default method

* Refactor Attachmentable concern and delay Paperclip's content-type spoof check

Check for content-type spoofing *after* setting the extension ourselves, this
fixes a regression with kt-paperclip's validations being more strict than
paperclip 6.0.0 and rejecting some Pleroma uploads because of unknown
extensions.

* Please CodeClimate

* Add audio/vorbis to the unreliable set

It doesn't correspond to a file format and thus has no extension associated.
2021-09-29 23:52:36 +02:00
Claire 1fcf310c60 Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Takeshi Umeda ba1bc51afd Fix unsupported video error message handling (#16581) 2021-08-08 15:28:57 +02:00
Claire ab4c1478b6 Change references to tootsuite/mastodon to mastodon/mastodon (#16491)
* Change references to tootsuite/mastodon to mastodon/mastodon

* Remove obsolete test fixture

* Replace occurrences of tootsuite/mastodon with mastodon/mastodon in CHANGELOG

And a few other places
2021-07-13 15:46:20 +02:00
Eugen Rochko 34dd743b58 Add ability to skip sign-in token authentication for specific users (#16427)
Remove "active within last two weeks" exception for sign in token requirement

Change admin reset password to lock access until the password is reset
2021-07-08 05:31:28 +02:00
Claire 45893a757e Add --by-uri option to `tootctl domains purge` (#16434)
Fixes #16410
2021-07-07 21:17:00 +02:00
Claire 36031e5e5c Update emoji codepoint mappings to v13.1 (#16352) 2021-06-03 16:08:07 +02:00
Eugen Rochko 2b38d0f349 Bump version to 3.4.1 (#16350) 2021-06-03 04:26:02 +02:00
Claire 273e1258ce Fix migration script not being able to run if it fails midway (#16312)
* Fix migration script not being able to run if it fails midway

* Fix old migration script

* Fix old migration script

* Refactor CorruptionError
2021-06-02 19:15:17 +02:00
Claire 2c3a1f32ad Fix account deletion sometimes failing because of optimistic locks (#16317)
* Fix account deletion sometimes failing because of optimistic locks

In some rare occasions[1], deleting accounts would fail with a
`StaleObjectError` exception.

Indeed, account deletion manually sets the `AccountStat` values without
handling cases where the optimistic locking on `AccountStat` would fail.

To my knowledge, with the rewrite of account counters in #15913, the
`DeleteAccountService` is now the only place that changes the counters in
a way that is not atomic.

Since in this specific case, we do not care about the previous values of the
account counters, it appears we don't need locking at all for this table
anymore.

[1]: https://discourse.joinmastodon.org/t/account-cant-be-deleted/3602

* Bump MAX_SUPPORTED_VERSION in maintenance script
2021-06-02 17:41:25 +02:00
Claire 7f6a00ed0a Fix `tootctl search deploy` on Ruby 3 (#16346)
Fixes #16344
2021-06-01 17:31:49 +02:00
Eugen Rochko 85740837c6 Bump version to 3.4.0 (#16239) 2021-05-16 23:55:07 +02:00
Eugen Rochko 0be04106b2 Add more checks to `repo:check_locales_files` (#16249) 2021-05-16 23:54:46 +02:00
Claire 2934b06a25 Update fix-duplicates maintenance script to support latest migrations (#16231)
* Update maintenance script to support latest database migrations

* Update Account#merge_with!
2021-05-12 23:19:44 +02:00
rinsuki 503d7feb99 Fix animated GIF generates animated thumbnail (#16216) 2021-05-11 19:15:11 +02:00
Eugen Rochko cafaf8e783 Bump version to 3.4.0rc2 (#16206) 2021-05-10 22:23:48 +02:00
Takeshi Umeda 00f4204d05 Add a Redis environment variable for sidekiq (#16188) 2021-05-09 10:40:17 +02:00
Eugen Rochko 0245174bae Bump version to 3.4.0rc1 (#16053) 2021-05-08 17:15:06 +02:00
Claire bb69774177 Fix older migrations on Ruby 3 (#16174) 2021-05-07 15:56:45 +02:00
Claire 86f5fad111 Add Ruby 3.0 support (#16046)
* Fix issues with POSIX::Spawn, Terrapin and Ruby 3.0

Also improve the Terrapin monkey-patch for the stderr/stdout issue.

* Fix keyword argument handling throughout the codebase

* Monkey-patch Paperclip to fix keyword arguments handling in validators

* Change validation_extensions to please CodeClimate

* Bump microformats from 4.2.1 to 4.3.1

* Allow Ruby 3.0

* Add Ruby 3.0 test target to CircleCI

* Add test for admin dashboard warnings

* Fix admin dashboard warnings on Ruby 3.0
2021-05-06 14:22:54 +02:00
Eugen Rochko 115cdee0ed Fix media redownload worker retrying on unexpected response codes (#16111) 2021-05-05 23:46:59 +02:00
Eugen Rochko 1c2b085a64 Fix media processing getting stuck on too much stdin/stderr (#16136)
* Fix media processing getting stuck on too much stdin/stderr

See thoughtbot/terrapin#5

* Remove dependency on paperclip-av-transcoder gem

* Remove dependency on streamio-ffmpeg gem

* Disable stdin on ffmpeg process
2021-05-05 19:44:01 +02:00
Takeshi Umeda 7bcd4a83f8 Fix how to change connection pool for rails 6 (#16158)
* Fix how to change connection pool for rails 6

* Fix to match the code elsewhere
2021-05-05 04:41:10 +02:00
Takeshi Umeda 465434aac8 Fix cache redis not being used (#16131) 2021-04-29 15:43:49 +02:00
Ikko Ashimine 02941921a6 Fix typo in db.rake (#16126)
occured -> occurred
2021-04-27 16:25:24 +02:00
Claire 89ef33df51 Add border to 🚲 emoji (#16035) 2021-04-13 23:43:51 +02:00
Eugen Rochko b0ba97e1c1 Remove dependency on pluck_each gem (#16012) 2021-04-12 03:35:58 +02:00
abcang 57d0cb0002 Fix DB connection pool settings in CLI (#15983) 2021-03-31 17:46:17 +02:00
Stanislas 9c85025a11 tootctl emoji import: case insensitive duplicate check (#15738) 2021-03-24 10:55:16 +01:00
Claire b2a89bf38e Update Mastodon to Rails 6.1 (#15910)
* Update devise-two-factor to unreleased fork for Rails 6 support

Update tests to match new `rotp` version.

* Update nsa gem to unreleased fork for Rails 6 support

* Update rails to 6.1.3 and rails-i18n to 6.0

* Update to unreleased fork of pluck_each for Ruby 6 support

* Run "rails app:update"

* Add missing ActiveStorage config file

* Use config.ssl_options instead of removed ApplicationController#force_ssl

Disabled force_ssl-related tests as they do not seem to be easily testable
anymore.

* Fix nonce directives by removing Rails 5 specific monkey-patching

* Fix fixture_file_upload deprecation warning

* Fix yield-based test failing with Rails 6

* Use Rails 6's index_with when possible

* Use ActiveRecord::Cache::Store#delete_multi from Rails 6

This will yield better performances when deleting an account

* Disable Rails 6.1's automatic preload link headers

Since Rails 6.1, ActionView adds preload links for javascript files
in the Links header per default.

In our case, that will bloat headers too much and potentially cause
issues with reverse proxies. Furhermore, we don't need those links,
as we already output them as HTML link tags.

* Switch to Rails 6.0 default config

* Switch to Rails 6.1 default config

* Do not include autoload paths in the load path
2021-03-24 10:44:31 +01:00
Claire edc41c6e17 Change mastodon:setup to not call assets:precompile in docker (#13942)
It appears assets are built during image build, and they shouldn't need
to be rebuilt, since we now have reproducible builds.
2021-03-24 10:37:24 +01:00
Claire 5c69eecacc Improve account counters handling (#15913)
* Improve account counters handling

* Use ActiveRecord::Base::sanitize_sql to pass values instead of interpolating them

Keep using string interpolation for `key` as it is safe and using
“ActiveRecord::Base::sanitize_sql_hash_for_assignment” would require stitching
bits of SQL in a way that is not more easily checked for safety.

* Add migration hook to catch PostgreSQL versions earlier than 9.5
2021-03-19 13:14:57 +01:00
Claire 7d180ee89c Remove MySQL-specific code from Mastodon::MigrationHelpers (#15924)
Mastodon::MigrationHelpers has been forked from Gitlab a long time ago, but
Mastodon has never supported using a MySQL database.

Removing MySQL support from Mastodon::MigrationHelpers makes it a little easier
to maintain. In particular, it removes code that would need updating with
Rails 6.
2021-03-19 13:14:40 +01:00