Commit Graph

33 Commits (84e332a9a1d449dd06d00df312037d0c760eb763)

Author SHA1 Message Date
Colin Mitchell 017ec81873 Add environment variable to configure sidekiq concurrency (#19589)
Co-authored-by: Effy Elden <effy@effy.space>
2022-12-15 16:38:37 +01:00
Eugen Rochko e6e139ac2b Change incoming activity processing to happen in `ingress` queue (#20264) 2022-11-10 14:21:51 +01:00
Claire 9a98169232 Change account deletion requests to spread out over time (#20222) 2022-11-09 14:08:19 +01:00
Eugen Rochko 7a482567f8 Add retention policy for cached content and media (#19232) 2022-09-27 03:08:19 +02:00
Eugen Rochko 7300beadd9 Remove digest e-mails (#17985)
* Remove digest e-mails

* Remove digest-related code
2022-08-25 23:38:22 +02:00
Eugen Rochko f4cab3ebad Change search indexing to use batches to minimize resource usage (#18451) 2022-05-18 23:29:14 +02:00
Eugen Rochko 1cd4518c29 Remove IP matching from e-mail domain blocks (#18190)
Clear out e-mail domain blocks created from automatically resolved DNS records
2022-04-29 23:27:03 +02:00
Eugen Rochko eabcc9f888 Change frequency of trends review notifications from 2 hours to 6 hours (#17850) 2022-03-22 11:58:25 +01:00
Eugen Rochko ea5bf21a1d Change e-mail domain blocks to block IPs dynamically (#17635)
* Change e-mail domain blocks to block IPs dynamically

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-24 17:28:23 +01:00
Eugen Rochko ad73becf3e Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire 1fcf310c60 Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Eugen Rochko 92b2d926bf Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
Takeshi Umeda a0c4783673 Fix to isolate the sidekiq process that runs the scheduler job (#15314) 2020-12-15 03:04:03 +01:00
Eugen Rochko df8874b24e Fix performance on instances list in admin UI (#15282)
- Reduce duplicate queries
- Remove n+1 queries
- Add accounts count to detailed view
- Add separate action log entry for updating existing domain blocks
2020-12-14 09:06:34 +01:00
Eugen Rochko 436266ca27 Change trending hashtags to not disappear instantly after midnight (#11712) 2019-09-02 18:11:13 +02:00
mayaeh 7d13da568e Add `tootctl preview_cards remove` (#11320)
* Add `tootctl preview_cards remove`

* fix code style

* Remove `Scheduler::PreviewCardsCleanupScheduler` file

* fix code style again
Add exclude case where image_file_name is blank

* Added a function to output confirmation if the specified number of days is less than 2 weeks
2019-07-28 13:48:19 +02:00
Eugen Rochko f9ed4354b1 Remove WebSub subscriptions (#11303) 2019-07-21 04:08:00 +02:00
Eugen Rochko ac853dca51 Add periodic removal of older thumbnails for preview cards (#11304) 2019-07-15 07:50:14 +02:00
Eugen Rochko 4931208dd8 Remove Salmon and PubSubHubbub (#11205)
* Remove Salmon and PubSubHubbub endpoints

* Add error when trying to follow OStatus accounts

* Fix new accounts not being created in ResolveAccountService
2019-07-06 23:26:16 +02:00
Eugen Rochko ae1aaa3b8a Add scheduled statuses (#9706)
Fix #340
2019-01-05 12:43:28 +01:00
Eugen Rochko ad0f37e924 Track historical space stats in PgHero to determine PostgreSQL growth (#8906) 2018-10-09 19:35:14 +02:00
Eugen Rochko 497092aee0 Add nginx and systemd templates (#8770)
So they can be copied during installation instead of looking
them up in the documentation

Make default sidekiq configuration use weighted queues

Remove deprecated docs directory
2018-09-24 16:46:05 +02:00
ThibG 56caf50db7 Update docker config and move some workers to different queues (#8345)
* Reorder docker-compose's sidekiq queues to match config/sidekiq.yml

* Make the ̀“mailers” queue higher priority than “pull”
2018-09-02 16:13:06 +02:00
Eugen Rochko f6884555d7 Account archive download (#6460)
* Fix #201: Account archive download

* Export actor and private key in the archive

* Optimize BackupService

- Add conversation to cached associations of status, because
  somehow it was forgotten and is source of N+1 queries
- Explicitly call GC between batches of records being fetched
  (Model class allocations are the worst offender)
- Stream media files into the tar in 1MB chunks
  (Do not allocate media file (up to 8MB) as string into memory)
- Use #bytesize instead of #size to calculate file size for JSON
  (Fix FileOverflow error)
- Segment media into subfolders by status ID because apparently
  GIF-to-MP4 media are all named "media.mp4" for some reason

* Keep uniquely generated filename in Paperclip::GifTranscoder

* Ensure dumped files do not overwrite each other by maintaing directory partitions

* Give tar archives a good name

* Add scheduler to remove week-old backups

* Fix code style issue
2018-02-21 23:21:32 +01:00
Eugen Rochko 67ee638c28 Move e-mail digest task to sidekiq, reduce workload, improve hint (#6252) 2018-01-15 04:34:28 +01:00
Patrick Figel dde43a9e32 Randomize sidekiq-scheduler cron schedule (#4980)
SubscriptionsScheduler in particular causes high load across the
entire fediverse at 5 AM UTC every day. Randomizing cron schedules
and/or adding a random delay is considered best practice to avoid
this issue.
2017-09-17 11:58:20 +02:00
sdukhovni 1fb5e3b009 Add scheduled worker to purge old user IPs (#4951)
* Add scheduled worker to purge old user IPs

* Use ruby 1.9 hash syntax
2017-09-14 22:26:38 +02:00
Eugen Rochko 0268b334ae Periodically remove expired PuSH subscribers (#4654) 2017-08-21 22:56:33 +02:00
Eugen Rochko 97dda48f70 Move clean up of unconfirmed users to sidekiq-scheduler (#4336)
* Move clean up of unconfirmed users to sidekiq-scheduler

* mastodon:daily is now deprecated
2017-07-24 16:17:55 +02:00
Eugen Rochko ad531f90b1 Bind web UI access tokens to sessions (#3940)
* Add overview of active sessions

* Better display of browser/platform name

* Improve how browser information is stored and displayed for sessions overview

* Fix test

* Fix #2347 - Bind web UI access token to session

When you logout, session also destroys the access token, so it's no longer
valid. If access token is destroyed some other way, the session is also
destroyed, requiring a re-login.

Fix #1681 - Add scheduler to remove revoked access tokens and grants

* Fix test
2017-06-25 23:51:32 +02:00
Eugen Rochko b51beb6486 Replace mastodon:media:clear and mastodon:feeds:clear rake tasks with (#3180)
sidekiq-scheduler jobs

Resolves #2495
2017-05-20 19:42:58 +02:00
Eugen Rochko c32e312061 More robust PuSH subscription refreshes (#2799)
* Fix #2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron

Fix an issue where / in domain would raise exception in TagManager#normalize_domain

PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through

Fix #2702 - Correct user agent header on outgoing http requests

* Add test for SubscribeService

* Extract #expiring_accounts into method

* Make mastodon:push:refresh no-op

* Queues are now defined in sidekiq.yml

* Queues are now in sidekiq.yml
2017-05-05 02:23:01 +02:00
Eugen Rochko 36f3da3cde Adjust down the number of Sidekiq threads to 5 (default of 25 is way too high) 2016-03-25 14:20:31 +01:00