Go to file
Kaspar V 47f0d7021e
refactor(vacuum statuses): reduce amount of db queries and load for each query - improve performance (#21487)
* refactor(statuses_vacuum): remove dead code - unused

Method is not called inside class and private.
Clean up dead code.

* refactor(statuses_vacuum): make retention_period present test explicit

This private method only hides functionality.
It is best practice to be as explicit as possible.

* refactor(statuses_vacuum): improve query performance

- fix statuses_scope having sub-select for Account.remote scope by
  `joins(:account).merge(Account.remote)`
- fix statuses_scope unnecessary use of `Status.arel_table[:id].lt`
  because it is inexplicit, bad practice and even slower than normal
  `.where('statuses.id < ?'`
- fix statuses_scope remove select(:id, :visibility) for having reusable
  active record query batches (no re queries)
- fix vacuum_statuses! to use in_batches instead of find_in_batches,
  because in_batches delivers a full blown active record query result,
  in stead of an array - no requeries necessary
- send(:unlink_from_conversations) not to perform another db query, but
  reuse the in_batches result instead.
- remove now obsolete remove_from_account_conversations method
- remove_from_search_index uses array of ids, instead of mapping
  the ids from an array - this should be more efficient
- use the in_batches scope to call delete_all, instead of running
  another db query for this - because it is again more efficient
- add TODO comment for calling models private method with send

* refactor(status): simplify unlink_from_conversations

- add `has_many through:` relation mentioned_accounts
- use model scope local instead of method call `Status#local?`
- more readable add account to inbox_owners when account.local?

* refactor(status): searchable_by way less sub selects

These queries all included a sub-select. Doing the same with a joins
should be more efficient.
Since this method does 5 such queries, this should be significant,
since it technically halves the query count.

This is how it was:

```ruby
[3] pry(main)> Status.first.mentions.where(account: Account.local, silent: false).explain
  Status Load (1.6ms)  SELECT "statuses".* FROM "statuses" WHERE "statuses"."deleted_at" IS NULL ORDER BY "statuses"."id" DESC LIMIT $1  [["LIMIT", 1]]
  Mention Load (1.5ms)  SELECT "mentions".* FROM "mentions" WHERE "mentions"."status_id" = $1 AND "mentions"."account_id" IN (SELECT "accounts"."id" FROM "accounts" WHERE "accounts"."domain" IS NULL) AND "mentions"."silent" = $2  [["status_id", 109382923142288414], ["silent", false]]
=> EXPLAIN for: SELECT "mentions".* FROM "mentions" WHERE "mentions"."status_id" = $1 AND "mentions"."account_id" IN (SELECT "accounts"."id" FROM "accounts" WHERE "accounts"."domain" IS NULL) AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
                                                    QUERY PLAN
------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.15..23.08 rows=1 width=41)
   ->  Seq Scan on accounts  (cost=0.00..10.90 rows=1 width=8)
         Filter: (domain IS NULL)
   ->  Index Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.15..8.17 rows=1 width=41)
         Index Cond: ((account_id = accounts.id) AND (status_id = '109382923142288414'::bigint))
         Filter: (NOT silent)
(6 rows)
```

This is how it is with this change:

```ruby
[4] pry(main)> Status.first.mentions.joins(:account).merge(Account.local).active.explain
  Status Load (1.7ms)  SELECT "statuses".* FROM "statuses" WHERE "statuses"."deleted_at" IS NULL ORDER BY "statuses"."id" DESC LIMIT $1  [["LIMIT", 1]]
  Mention Load (0.7ms)  SELECT "mentions".* FROM "mentions" INNER JOIN "accounts" ON "accounts"."id" = "mentions"."account_id" WHERE "mentions"."status_id" = $1 AND "accounts"."domain" IS NULL AND "mentions"."silent" = $2  [["status_id", 109382923142288414], ["silent", false]]
=> EXPLAIN for: SELECT "mentions".* FROM "mentions" INNER JOIN "accounts" ON "accounts"."id" = "mentions"."account_id" WHERE "mentions"."status_id" = $1 AND "accounts"."domain" IS NULL AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
                                                    QUERY PLAN
------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.15..23.08 rows=1 width=41)
   ->  Seq Scan on accounts  (cost=0.00..10.90 rows=1 width=8)
         Filter: (domain IS NULL)
   ->  Index Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.15..8.17 rows=1 width=41)
         Index Cond: ((account_id = accounts.id) AND (status_id = '109382923142288414'::bigint))
         Filter: (NOT silent)
(6 rows)
```
2022-11-27 20:41:18 +01:00
.circleci Revamp post filtering system (#18058) 2022-06-28 09:42:13 +02:00
.devcontainer Fix auto detect language for translate service (#19244) 2022-09-27 23:33:56 +02:00
.github Use buildx functions for faster build (#20692) 2022-11-17 11:01:16 +01:00
app refactor(vacuum statuses): reduce amount of db queries and load for each query - improve performance (#21487) 2022-11-27 20:41:18 +01:00
bin Update Mastodon to Rails 6.1 (#15910) 2021-03-24 10:44:31 +01:00
chart Fix the command to create the admin user (#19827) 2022-11-24 21:32:03 +01:00
config Add logging for Rails cache timeouts (#21667) 2022-11-27 20:37:37 +01:00
db Fix featured tags not saving preferred casing (#19732) 2022-11-04 16:08:29 +01:00
dist Fix nginx location matching (#20198) 2022-11-09 04:12:57 +01:00
lib Add logging for Rails cache timeouts (#21667) 2022-11-27 20:37:37 +01:00
log Initial commit 2016-02-20 22:53:20 +01:00
public Add assets from Twemoji 14.0 (#19733) 2022-11-04 16:08:41 +01:00
spec Fix attachments of edited statuses not being fetched (#21565) 2022-11-27 20:39:05 +01:00
streaming Fix filters from other users being used in the streaming service (#20719) 2022-11-15 02:09:58 +01:00
vendor Increase files checked by ESLint (#9705) 2019-01-04 11:28:38 +01:00
.browserslistrc Add ES6 compatibility to browserslist (#18519) 2022-05-26 20:29:28 +02:00
.buildpacks Remove nodejs buildpack from buildpacks (#14364) 2020-07-20 09:48:10 +02:00
.codeclimate.yml Replace from Code Climate to Super-Linter (#18587) 2022-06-01 19:22:35 +02:00
.deepsource.toml Update .deepsource.toml (#15753) 2021-02-19 09:53:25 +01:00
.dockerignore Update .dockerignore/.gitignore for #16947 etc (#17099) 2021-12-06 18:04:04 +01:00
.editorconfig
.env.production.sample ip_cleanup_scheduler: Make IP and session retention configurable (#18757) 2022-07-07 03:14:28 +02:00
.env.test Bump webpacker from 4.0.7 to 4.2.0 (#12416) 2019-11-20 17:56:11 +01:00
.env.vagrant Add a default DB_HOST to .env.vagrant for enable the streaming (#14030) 2020-06-15 01:57:06 +02:00
.eslintignore Increase files checked by ESLint (#9705) 2019-01-04 11:28:38 +01:00
.eslintrc.js Replace from Code Climate to Super-Linter (#18587) 2022-06-01 19:22:35 +02:00
.foreman
.gitattributes
.gitignore Helm update readme.md (#20154) 2022-11-10 19:09:54 +01:00
.haml-lint.yml
.nanoignore
.nvmrc NodeJS 14 support - circleci/docker/.nvmrc (#16163) 2021-08-10 22:56:13 +02:00
.prettierignore Format JSON and YAML using Prettier (#17823) 2022-03-21 04:46:11 +01:00
.prettierrc.js Format JSON and YAML using Prettier (#17823) 2022-03-21 04:46:11 +01:00
.profile
.rspec
.rubocop.yml Fix style for hashes (#20518) 2022-11-17 11:05:39 +01:00
.ruby-gemset Make Mastodon use its own gemset (#17858) 2022-08-13 15:44:34 +02:00
.ruby-version Bump Ruby version from 3.0.3 to 3.0.4 (#18028) 2022-08-15 04:39:58 +02:00
.slugignore
.yarnclean Replace from scss-lint to sass-lint (#10958) 2019-06-04 17:23:18 +02:00
AUTHORS.md Update AUTHORS.md (#20630) 2022-11-14 08:33:24 +01:00
Aptfile Heroku fix (#19807) 2022-11-05 22:57:58 +01:00
CHANGELOG.md Bump version to 4.0.2 (#20725) 2022-11-15 03:57:18 +01:00
CODE_OF_CONDUCT.md
CONTRIBUTING.md Note that CircleCI auth may be required to run PR pipelines (#20371) 2022-11-11 02:54:02 +01:00
Capfile
Dockerfile Add missing procps package to Dockerfile (#21028) 2022-11-22 05:52:18 +01:00
FEDERATION.md Add FEDERATION.md (#17029) 2021-11-23 00:15:31 +01:00
Gemfile Bump capybara from 3.37.1 to 3.38.0 (#20036) 2022-11-26 13:51:53 +09:00
Gemfile.lock Bump capybara from 3.37.1 to 3.38.0 (#20036) 2022-11-26 13:51:53 +09:00
LICENSE
Procfile fix: make Procfile compatible with herokuish (#12685) 2020-08-29 19:00:30 +02:00
Procfile.dev Set RAILS_ENV in Procfile.dev (#15502) 2021-01-07 09:30:12 +01:00
README.md Drop support for Ruby v2.5 and Node.js v12 (#18323) 2022-06-16 22:09:58 +02:00
Rakefile
SECURITY.md Change e-mail in SECURITY.md (#20384) 2022-11-11 05:26:43 +01:00
Vagrantfile Remove protobuf dependencies (#17539) 2022-02-14 16:08:02 +01:00
app.json Make enable_starttls configurable by envvars (#20321) 2022-11-10 21:06:21 +01:00
babel.config.js Remove dependency for @babel/plugin-proposal-class-properties (#16155) 2021-05-04 23:13:04 +02:00
config.ru
crowdin.yml Prevent translations from accidentally becoming completely different from source strings on Crowdin without translators noticing. (#17085) 2021-12-01 19:03:35 +01:00
docker-compose.yml Update docker-compose.yml (#19063) 2022-08-27 17:40:46 +02:00
ide-helper.js Add JS IDE helper (#13012) 2020-04-27 17:19:53 +02:00
jest.config.js Bump jest from 27.5.1 to 28.0.3 (#18280) 2022-05-04 14:06:49 +09:00
jsconfig.json Replace `CancelToken` to `AbortSignal` (#19352) 2022-10-14 03:16:37 +02:00
package.json Bump sass from 1.55.0 to 1.56.1 (#20681) 2022-11-26 14:07:52 +09:00
postcss.config.js Increase files checked by ESLint (#9705) 2019-01-04 11:28:38 +01:00
priv-config TOR federation (#7875) 2018-06-26 20:34:12 +02:00
scalingo.json Make enable_starttls configurable by envvars (#20321) 2022-11-10 21:06:21 +01:00
stylelint.config.js Replace from Code Climate to Super-Linter (#18587) 2022-06-01 19:22:35 +02:00
yarn.lock Bump sass from 1.55.0 to 1.56.1 (#20681) 2022-11-26 14:07:52 +09:00

README.md

Mastodon

GitHub release Build Status Code Climate Crowdin Docker Pulls

Mastodon is a free, open-source social network server based on ActivityPub where users can follow friends and discover new ones. On Mastodon, users can publish anything they want: links, pictures, text, video. All Mastodon servers are interoperable as a federated network (users on one server can seamlessly communicate with users from another one, including non-Mastodon software that implements ActivityPub)!

Click below to learn more in a video:

Screenshot

Navigation

Features

No vendor lock-in: Fully interoperable with any conforming platform

It doesn't have to be Mastodon; whatever implements ActivityPub is part of the social network! Learn more

Real-time, chronological timeline updates

Updates of people you're following appear in real-time in the UI via WebSockets. There's a firehose view as well!

Media attachments like images and short videos

Upload and view images and WebM/MP4 videos attached to the updates. Videos with no audio track are treated like GIFs; normal videos loop continuously!

Safety and moderation tools

Mastodon includes private posts, locked accounts, phrase filtering, muting, blocking and all sorts of other features, along with a reporting and moderation system. Learn more

OAuth2 and a straightforward REST API

Mastodon acts as an OAuth2 provider, so 3rd party apps can use the REST and Streaming APIs. This results in a rich app ecosystem with a lot of choices!

Deployment

Tech stack:

  • Ruby on Rails powers the REST API and other web pages
  • React.js and Redux are used for the dynamic parts of the interface
  • Node.js powers the streaming API

Requirements:

  • PostgreSQL 9.5+
  • Redis 4+
  • Ruby 2.6+
  • Node.js 14+

The repository includes deployment configurations for Docker and docker-compose as well as specific platforms like Heroku, Scalingo, and Nanobox. The standalone installation guide is available in the documentation.

A Vagrant configuration is included for development purposes. To use it, complete following steps:

  • Install Vagrant and Virtualbox
  • Install the vagrant-hostsupdater plugin: vagrant plugin install vagrant-hostsupdater
  • Run vagrant up
  • Run vagrant ssh -c "cd /vagrant && foreman start"
  • Open http://mastodon.local in your browser

Contributing

Mastodon is free, open-source software licensed under AGPLv3.

You can open issues for bugs you've found or features you think are missing. You can also submit pull requests to this repository or submit translations using Crowdin. To get started, take a look at CONTRIBUTING.md. If your contributions are accepted into Mastodon, you can request to be paid through our OpenCollective.

IRC channel: #mastodon on irc.libera.chat

License

Copyright (C) 2016-2022 Eugen Rochko & other Mastodon contributors (see AUTHORS.md)

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.