Commit Graph

669 Commits (91dd5ce8b75376b348528b6421e1c936717bea8e)

Author SHA1 Message Date
Eugen Rochko 5a7219e532 Refactor formatter (#17828)
* Refactor formatter

* Move custom emoji pre-rendering logic to view helpers

* Move more methods out of Formatter

* Fix code style issues

* Remove Formatter

* Add inline poll options to RSS feeds

* Remove unused helper method

* Fix code style issues

* Various fixes and improvements

* Fix test
2022-03-26 02:53:34 +01:00
Eugen Rochko 88f93d03fe Fix edits with no actual changes being allowed (#17843)
* Fix edits with no actual changes being allowed locally

* Fix edits with no actual changes being allowed through ActivityPub

* Fix false positive changes caused by description processing in model

* Fix not recording poll expiration update

* Fix test

* Revert changes to ProcessStatusUpdateService

* Various fixes and improvements

* Fix code style issues

* Various changes and improvements

* Add guard clause
2022-03-26 00:38:44 +01:00
Claire d17dfe4e87 Add admin dashboard checks for Elasticsearch version (#17863) 2022-03-25 21:00:59 +01:00
Eugen Rochko 79d186ffa8 Add workaround for YouTube Shorts links (#17869)
* Add workaround for YouTube Shorts links

* Update link_details_extractor_spec.rb
2022-03-25 19:31:35 +01:00
Claire cf4af2faba Fix searching already-known converted activities by URL (#17814)
Fixes #17813
2022-03-17 00:46:49 +01:00
Eugen Rochko dcbc83c9b3 Fix not updating a status when newer version is fetched manually (#17745) 2022-03-12 09:11:36 +01:00
Eugen Rochko 9c2275d15f Change how changes to media attachments are stored for edits (#17696)
* Change how changes to media attachments are stored for edits

Fix not being able to re-order media attachments

* Fix not broadcasting updates when polls/media is changed through ActivityPub

* Various fixes and improvements

* Update app/models/report.rb

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

* Add tracking of media attachment description changes

* Change poll in status edit to have a structure closer to the real one

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-09 09:06:17 +01:00
Eugen Rochko 11e848bf95 Change design of federation pages in admin UI (#17704)
* Change design of federation pages in admin UI

* Fix query performance in instance media attachments measure

* Fix reblogs being included in instance languages dimension
2022-03-09 08:52:32 +01:00
Eugen Rochko e3d3b4ae21 Add `/api/v1/accounts/familiar_followers` to REST API (#17700)
* Add `/api/v1/accounts/familiar_followers` to REST API

* Change hide network preference to be stored consistently for local and remote accounts

* Add dummy classes to migration

* Apply suggestions from code review

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-07 09:36:47 +01:00
Eugen Rochko 3eaa7e21e3 Fix duplicate notifications being possible after poll expiration (#17697) 2022-03-04 01:06:33 +01:00
Eugen Rochko 799a812dbe Fix infinite re-forwarding of update activities temporarily (#17698) 2022-03-04 01:00:13 +01:00
Eugen Rochko ba969db9c8 Fix returning empty `<p>` tag for blank account `note` in REST API (#17687) 2022-03-03 06:45:46 +01:00
Eugen Rochko 4d0a2968d9 Change authorized applications page (#17656)
* Change authorized applications page

* Hide revoke button for superapps and suspended accounts

* Clean up db/schema.rb
2022-03-01 16:48:58 +01:00
Eugen Rochko 9d138baa73 Fix status updates not being forwarded like deletes through ActivityPub (#17648)
Fix #17521
2022-02-26 17:51:59 +01:00
Eugen Rochko e6d2b07ec1 Add trending statuses (#17431)
* Add trending statuses

* Fix dangling items with stale scores in localized sets

* Various fixes and improvements

- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction

* Add tests for trending statuses filtering behaviour

* Fix not applying filtering scope in controller
2022-02-25 00:34:14 +01:00
luzpaz b97e7364f3 Fix various typos (#17621)
Found via `codespell -q 3 -S ./CHANGELOG.md,./AUTHORS.md,./config/locales,./app/javascript/mastodon/locales -L ba,keypair,medias,ro`
2022-02-22 20:14:17 +01:00
Claire f499f086a9 Fix some media attachments being converted with too high framerates (#17619)
Video files with variable framerates are converted to constant framerate videos
and the output framerate picked by ffmpeg is based on the original file's
container framerate (which can be different from the average framerate).

This means that an input video with variable framerate with about 30 frames per
second on average, but a maximum of 120 fps will be converted to a constant 120
fps file, which won't be processed by other Mastodon servers.

This commit changes it so that input files with VFR and a maximum framerate
above the framerate threshold are converted to VFR files with the maximum frame
rate enforced.
2022-02-22 17:11:22 +01:00
Eugen Rochko 0fbd040aa8 Add caching layer to metrics (#17617) 2022-02-22 15:27:08 +01:00
Claire efb2377148 Change relays handling to not record boosts (#17571)
* Change relays handling to not record boosts

* Update tests
2022-02-16 14:36:44 +01:00
Claire 4d49bb4b52 Fix performance of server-side filtering (#17575)
Fixes #17567
2022-02-16 14:28:45 +01:00
Jeong Arm cf8fee1379 Add `from:` query operator to search syntax (#16526)
* Add 'by:userhandle' parameter to search api

* Use search syntax for "by" prefix

* Codeclimate

* Use 'from' instead of 'by'
2022-02-14 00:17:09 +01:00
Eugen Rochko 6732217dd0 Change languages to be listed under standard instead of native name in admin UI (#17485) 2022-02-09 04:15:38 +01:00
Eugen Rochko 948da1a958 Add edit history to web UI (#17390)
* Add edit history to web UI

* Change history reducer to store items per status

* Fix missing loading prop
2022-02-09 01:17:07 +01:00
Eugen Rochko a44cc7931a Remove language detection through cld3 (#17478)
* Remove language detection through cld3

* Update app/helpers/languages_helper.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-08 02:41:17 +01:00
Eugen Rochko 73a585dd64 Fix structured data parsing from links choking on bad data (#17403)
* Fix structured data parsing from links choking on bad data

- Fix og:url meta tag being prioritized over canonical link tag
- Fix structured data parsing choking on commented-out CDATA declarations
- Fix HTML entities in title, description, provider_name, author_name
- Change structured data parsing to attempt every JSON-LD script tag

* Remove unnecessary slash escapes from CDATA regex pattern
2022-02-07 18:16:31 +01:00
Claire b891d09d2f Compact JSON-LD signed incoming activities (#17426)
Co-authored-by: Puck Meerburg <puck@puck.moe>
2022-02-03 14:07:29 +01:00
Claire 33ea1c9008 Fix Sidekiq warnings about JSON serialization (#17381)
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
Claire 41d64ee271 Change `percent` to `rate` in retention metrics API (#16910) 2022-01-23 16:01:25 +01:00
Eugen Rochko 06b698a723 Add support for editing for published statuses (#16697)
* Add support for editing for published statuses

* Fix references to stripped-out code

* Various fixes and improvements

* Further fixes and improvements

* Fix updates being potentially sent to unauthorized recipients

* Various fixes and improvements

* Fix wrong words in test

* Fix notifying accounts that were tagged but were not in the audience

* Fix mistake
2022-01-19 22:37:27 +01:00
Eugen Rochko 2fb76550a9 Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
Claire 5aade2baac Add support for private pinned posts (#16954)
* Add support for private pinned toots

* Allow local user to pin private toots

* Change wording to avoid "direct message"
2022-01-17 00:49:55 +01:00
Eugen Rochko 9a8784829f Fix warnings on Rails boot (#16946) 2021-12-27 00:47:20 +01:00
Jeong Arm cb22372cd7 Fix server graph on admin/tags/:id (#17066) 2021-11-28 23:13:07 +01:00
Eugen Rochko 4375813ea7 Remove Keybase integration (#17045) 2021-11-26 05:58:18 +01:00
Eugen Rochko ad73becf3e Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire 701472d1fc Fix confusing error when webfinger request returns empty document (#16986)
For some reason, some misconfigured servers return an empty document when
queried over webfinger. Since an empty document does not lead to a parse
error, the error is not caught properly and triggers uncaught exceptions
later on.

This PR fixes that by immediately erroring out with `Webfinger::Error` on
getting an empty response.
2021-11-14 21:55:40 +01:00
Eugen Rochko 1bf6ec3325 Fix no link previews being generated for pages with invalid structured data (#16979)
Fix #16955
2021-11-13 23:07:13 +01:00
Claire fdfcc4fcb7 Fix OpenGraph/LinkedData embeds having incorrectly-generated iframes (#16978) 2021-11-13 03:30:27 +01:00
Eugen Rochko 4dc87ffc06 Add support for structured data and more OpenGraph tags to link cards (#16938)
Save preview cards under their canonical URL

Increase max redirects to follow from 2 to 3
2021-11-05 23:23:05 +01:00
Claire bf1edcb0ef Fix retention rate computation (#16854) 2021-10-14 22:20:37 +02:00
Eugen Rochko cad446611c Add graphs and retention metrics to admin dashboard (#16829) 2021-10-14 20:44:59 +02:00
Claire 7d0089033f Switch from unmaintained paperclip to kt-paperclip (#16724)
* Switch from unmaintained paperclip to kt-paperclip

* Drop some compatibility monkey-patches not required by kt-paperclip

* Drop media spoof check monkey-patching

It's broken with kt-paperclip and hopefully it won't be needed anymore

* Fix regression introduced by paperclip 6.1.0

* Do not rely on pathname to call FastImage

* Add test for ogg vorbis file with cover art

* Add audio/vorbis to the accepted content-types

This seems erroneous as this would be the content-type for a vorbis stream
without an ogg container, but that's what the `marcel` gem outputs, so…

* Restore missing for_as_default method

* Refactor Attachmentable concern and delay Paperclip's content-type spoof check

Check for content-type spoofing *after* setting the extension ourselves, this
fixes a regression with kt-paperclip's validations being more strict than
paperclip 6.0.0 and rejecting some Pleroma uploads because of unknown
extensions.

* Please CodeClimate

* Add audio/vorbis to the unreliable set

It doesn't correspond to a file format and thus has no extension associated.
2021-09-29 23:52:36 +02:00
Eugen Rochko ce84967ee4 Change routing paths to use usernames in web UI (#16171) 2021-09-26 05:46:13 +02:00
Claire 067d10d664 Fix addressing of remote groups' followers (#16700)
Fixes #16699
2021-09-08 23:33:23 +02:00
Takeshi Umeda 15a56e3717 Fix invalid blurhash handling in Create activity (#16583) 2021-08-09 13:33:19 +02:00
Claire b5e603c6ad Add logging of S3-related errors (#16381) 2021-07-21 18:34:39 +02:00
Claire 9c1d3086af Fix inefficiencies in auto-linking code (#16506)
The auto-linking code basically rewrote the whole string escaping non-ascii
characters in an inefficient way, and building a full character offset map
between the unescaped and escaped texts before sending the contents to
TwitterText's extractor.

Instead of doing that, this commit changes the TwitterText regexps to include
valid IRI characters in addition to valid URI characters.
2021-07-15 15:56:58 +02:00
Claire 45903ae80a Fix some RedisLocks auto-releasing too fast (#16276)
* Fix Delete and Create-related locks expiring too fast

Fixes #16238

By default, RedisLock expires after 10 seconds, which may not be enough to
process statuses, especially when those have attached media files.

This commit extends those 10 seconds to 15 minutes, which should be plenty
enough to handle any status, while being short enough to not waste many
sidekiq job retries in the exceedingly rare case in which a sidekiq process
would crash when processing a `Create` or `Delete`.

* Fix other RedisLock autorelease durations

Fixes #15645

- things that only perform a few simple database queries (e.g. finding and
  saving a record) have been left unchanged, so they'll still use the default
  10s duration
- things that perform significantly more complex database queries have been
  changed to a 5 minutes timeout
- things that perform multiple HTTP queries have been changed to a 15 minutes
  timeout
2021-05-19 23:52:08 +02:00
abcang 15de844953 Fix rubocop warning (#16214) 2021-05-11 14:19:01 +02:00
Claire 5523279893 Fix blocking someone not clearing up list feeds (#16205) 2021-05-10 17:31:55 +02:00