Commit Graph

85 Commits (0687eb20dafb83d7a14d1a894a33733a57543100)

Author SHA1 Message Date
Brian Campbell baf43ec2d5 Fix idempotency when database writes are slow (#21840)
There is an idempotency key generated by clients when authoring a post,
and stored in Redis, to ensure that if a user or client retries posting
the same status, we don't get a duplicate.

Hachyderm.io has been experiencing some filesystem and database
performance issues, causing database writes to be slow. This can mean
that there are successful posts, but the reverse proxy returns 504
Gateway Timeout before the idempotency status has been updated; users or
clients who retry (such as Tusky which retries automatically, see
tuskyapp/Tusky#2951) can re-try the same post with the same idempotency
key before it has actually been recorded in Redis, leading to duplicate
posts.

To address this issue, move all of the database updates after the
initial transaction that creates the status into the
`postprocess_status!` method, so we can insert the idempotency key
immediately after the status has been created, significantly reducing
the window in which the status could be created but the idempotency key
not yet stored.

Note: this has not yet been tested; I'm submitting this PR for
discussion and to offer to the Hachyderm.io admins to try out to fix the
multiple posting problem.

Co-authored-by: Brian Campbell <brcampbell@beta.team>
2022-12-15 18:08:40 +01:00
Eugen Rochko 9c2275d15f Change how changes to media attachments are stored for edits (#17696)
* Change how changes to media attachments are stored for edits

Fix not being able to re-order media attachments

* Fix not broadcasting updates when polls/media is changed through ActivityPub

* Various fixes and improvements

* Update app/models/report.rb

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

* Add tracking of media attachment description changes

* Change poll in status edit to have a structure closer to the real one

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-09 09:06:17 +01:00
Eugen Rochko aff497a87a Fix invalid language resulting in no fallback being set on statuses (#17722) 2022-03-09 08:46:05 +01:00
Eugen Rochko a44cc7931a Remove language detection through cld3 (#17478)
* Remove language detection through cld3

* Update app/helpers/languages_helper.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-08 02:41:17 +01:00
Eugen Rochko ad73becf3e Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire 4a813ec686 Fix scheduled statuses decreasing statuses counts (#16791)
* Add tests

* Fix scheduled statuses decreasing statuses counts

Fixes #16774
2021-10-14 19:59:21 +02:00
ThibG 54da8c4337 Fix media not being marked sensitive when client sets a CW but no text (#13277)
Mastodon enforces the “sensitive” flag on media attachments whenever a toot
is posted with a Content Warning. However, it does so *after* potentially
converting the Content Warning to toot text (when there is no toot text),
which leads to inconsistent and surprising behavior for API clients.
This commit fixes this inconsistency.
2020-03-25 22:40:58 +01:00
Eugen Rochko f459919552 Change local media attachments to perform heavy processing asynchronously (#13210)
Fix #9106
2020-03-08 23:56:18 +01:00
ThibG 852958d725 Fix public posts from silenced accounts not being changed to unlisted visibility (#13096) 2020-03-08 16:11:49 +01:00
Eugen Rochko a4668d994b Add specific rate limits for posting and following (#13172) 2020-03-08 15:17:39 +01:00
ThibG f088e090e4 Add voters count support (#11917)
* Add voters count to polls

* Add ActivityPub serialization and parsing of voters count

* Add support for voters count in WebUI

* Move incrementation of voters count out of redis lock

* Reword “voters” to “people”
2019-09-29 22:58:01 +02:00
Eugen Rochko 4931208dd8 Remove Salmon and PubSubHubbub (#11205)
* Remove Salmon and PubSubHubbub endpoints

* Add error when trying to follow OStatus accounts

* Fix new accounts not being created in ResolveAccountService
2019-07-06 23:26:16 +02:00
Eugen Rochko d7e2f554fb Add audio uploads (#11123)
* Add audio uploads

Fix #4827

Accept uploads of OGG, WAV, FLAC, OPUS and MP3 files, and converts
them to OGG. Media attachments get a new `audio` type. In the UI,
audio uploads are displayed identically to video uploads.

* Improve code style
2019-06-19 23:42:38 +02:00
ThibG f1a3135809 Record account suspend/silence time and keep track of domain blocks (#10660)
* Record account suspend/silence time and keep track of domain blocks

* Also unblock users who were suspended/silenced before dates were recorded

* Add tests

* Keep track of suspending date for users suspended through the CLI

* Show accurate number of accounts that would be affected by unsuspending an instance

* Change migration to set silenced_at and suspended_at

* Revert "Also unblock users who were suspended/silenced before dates were recorded"

This reverts commit a015c65d2d1e28c7b7cfab8b3f8cd5fb48b8b71c.

* Switch from using suspended and silenced to suspended_at and silenced_at

* Add post-deployment migration script to remove `suspended` and `silenced` columns

* Use Account#silence! and Account#suspend! instead of updating the underlying property

* Add silenced_at and suspended_at migration to post-migration

* Change account fabricator to translate suspended and silenced attributes

* Minor fixes

* Make unblocking domains always retroactive
2019-05-14 19:05:02 +02:00
Eugen Rochko 364e49ae1f Rename :poll to :preloadable_poll and :owned_poll to :poll on Status (#10401)
Also, fix some n+1 queries

Resolve #10365
2019-03-28 04:44:59 +01:00
Eugen Rochko 0b6bcb9384 Fix detailed poll validation errors not being returned in the API (#10261)
No more "Owned poll is invalid"
2019-03-15 13:36:38 +01:00
ThibG 11ce444b8b Support pushing and receiving updates to poll tallies (#10209)
* Process incoming poll tallies update

* Send Update on poll vote

* Do not send Updates for a poll more often than once every 3 minutes

* Include voters in people to notify of results update

* Schedule closing poll worker on poll creation

* Add new notification type for ending polls

* Add front-end support for ended poll notifications

* Fix UpdatePollSerializer

* Fix Updates not being triggered by local votes

* Fix tests failure

* Fix web push notifications for closing polls

* Minor cleanup

* Notify voters of both remote and local polls when those close

* Fix delivery of poll updates to mentioned accounts and voters
2019-03-11 00:49:31 +01:00
ThibG ad11260825 Correctly make polls and media mutually exclusive (#10141) 2019-03-04 00:40:21 +01:00
Eugen Rochko 4a3acdc916 Add polls (#10111)
* Add polls

Fix #1629

* Add tests

* Fixes

* Change API for creating polls

* Use name instead of content for votes

* Remove poll validation for remote polls

* Add polls to public pages

* When updating the poll, update options just in case they were changed

* Fix public pages showing both poll and other media
2019-03-03 22:18:23 +01:00
ysksn 9cad360caa Create Redisable#redis (#9633)
* Create Redisable

* Use #redis instead of Redis.current
2019-02-02 19:11:38 +01:00
ThibG b1c76ef610 Prevent posting toots with media attachments from someone else (#9921) 2019-01-26 23:59:39 +01:00
ThibG a238beb870 Fix scheduled toot with media immediately creating a toot (#9894)
* Add test for not persisting status when attaching media to scheduled toot

* Prevent status used for validation from being persisted to the database

Fixes #9893

Thanks to tateisu for the help investigating this.
2019-01-21 20:03:04 +01:00
Eugen Rochko 17a8c10a03 Fix error when `scheduled_at` attribute is malformed (#9866) 2019-01-19 22:01:01 +01:00
Eugen Rochko bf1ee883d8 Fix wrong param name in scheduled statuses and return params in API (#9725)
The database column and API param are called in_reply_to_id, not
in_reply_to_status_id, so it makes no sense to encode it that way
2019-01-06 12:03:27 +01:00
Eugen Rochko ae1aaa3b8a Add scheduled statuses (#9706)
Fix #340
2019-01-05 12:43:28 +01:00
ThibG 0244534700 Avoid duplicate work by merging ReplyDistributionWorker into DistributionWorker (#9660) 2018-12-30 19:00:04 +01:00
ThibG a2492a89fb Limit maximum visibility of local silenced users to unlisted (#9583)
Fixes #9580
2018-12-24 19:06:14 +01:00
Eugen Rochko 38f413e583 Fix activity:interactions counter to count all interactions (#8037) 2018-07-16 18:35:43 +02:00
Eugen Rochko e57d9177dc Fix check for PotentialFriendshipTracker when replying (#7933) 2018-07-03 05:00:44 +02:00
Eugen Rochko aa4e75166b Re-add follow recommendations API (#7918)
* Re-add follow recommendations API

    GET /api/v1/suggestions

Removed in 30f9e9e624 due to Neo4J
dependency. The algorithm uses triadic closures, takes into account
suspensions, blocks, mutes, domain blocks, excludes locked and moved
accounts, and prefers more recently updated accounts.

* Track interactions with people you don't follow

Replying to, favouriting and reblogging someone you're not following
will make them show up in follow recommendations. The interactions
have different weights:

- Replying is 1
- Favouriting is 10 (decidedly positive interaction, but private)
- Reblogging is 20

Following them, muting or blocking will remove them from the list,
obviously.

* Remove triadic closures, ensure potential friendships are trimmed
2018-07-03 01:47:56 +02:00
Eugen Rochko f91fa10804 Allow selecting default posting language instead of auto-detect (#7828)
* Allow selecting default posting language instead of auto-detect

* Enable default language setting in credentials API

* Fix form saving
2018-06-17 18:57:31 +02:00
Eugen Rochko 681ac8580a Remove placeholder text for media-only toots (#7806)
Ref: 0c2a08adeb

This breaks compatibility with pre-2.3.0 Mastodon and older
software, but at the time of writing the network is >80% above
that version.

Compatibility broken only for toots with no text.
2018-06-15 15:51:51 +02:00
Akihiko Odaki 3114ac9655 Do not mark remote status sensitive even if spoiler text is present (#7395)
Old statuses and statuses from Pawoo, which runs a modified version of
Mastodon, may not have been marked sensitive even if spoiler text is
present.

Such statuses are still not marked sensitve if they are local or
arrived before version upgrade. Marking recently fetched remote status
sensitive contradicts the behavior.

Considering what people expected when they authored such statuses, this
change removes the sensitivity enforcement.
2018-06-04 04:46:14 +02:00
abcang e6c952476c Call media.present? because media may be nil (#7474) 2018-05-14 12:51:53 +02:00
ThibG 10be4c496c Fix hashtags not being federated on mentions (fixes #6900) (#7406) 2018-05-08 03:36:59 +02:00
Eugen Rochko b890e5830c Support explicitly supplying language code for status via REST API (#7389) 2018-05-07 09:30:53 +02:00
Renato "Lond" Cerqueira ddbc1572f2 When creating status, if no sensitive status is given, use default (#7057)
Clients using the API that do not provide the sensitive flag are always
posting with false sensitive option.
2018-04-09 23:02:42 +02:00
Eugen Rochko 0277ca9a0c For now, put a "." into no-text statuses with media for backcompat (#6691) 2018-03-08 08:20:49 +01:00
Eugen Rochko 0c2a08adeb Remove text requirement when media attached from statuses (#6672) 2018-03-07 08:28:52 +01:00
Yamagishi Kazutoshi fba46b6072 Using double splat operator (#5859) 2017-12-06 11:41:57 +01:00
ThibG b41ed24f74 Instantiate service classes for each call (fixes #5540) (#5543) 2017-10-27 19:08:30 +02:00
Akihiko Odaki d4a1ddd46a Fix filterable_languages method of SettingsHelper (#4966) 2017-09-16 14:59:41 +02:00
Eugen Rochko cc4bc2709d Fix #4850 - When visibility missing from API call to toot, fallback to user preference (#4861) 2017-09-09 02:02:29 +02:00
Eugen Rochko ca76d11434 Forward ActivityPub creates that reply to local statuses (#4709)
* Forward ActivityPub creates that reply to local statuses

* Fix test

* Fix wrong signers
2017-08-30 15:37:02 +02:00
Eugen Rochko 5516767c75 ActivityPub delivery (#4566)
* Deliver ActivityPub Like

* Deliver ActivityPub Undo-Like

* Deliver ActivityPub Create/Announce activities

* Deliver ActivityPub creates from mentions

* Deliver ActivityPub Block/Undo-Block

* Deliver ActivityPub Accept/Reject-Follow

* Deliver ActivityPub Undo-Follow

* Deliver ActivityPub Follow

* Deliver ActivityPub Delete activities

Incidentally fix #889

* Adjust BatchedRemoveStatusService for ActivityPub

* Add tests for ActivityPub workers

* Add tests for FollowService

* Add tests for FavouriteService, UnfollowService and PostStatusService

* Add tests for ReblogService, BlockService, UnblockService, ProcessMentionsService

* Add tests for AuthorizeFollowService, RejectFollowService, RemoveStatusService

* Add tests for BatchedRemoveStatusService

* Deliver updates to a local account to ActivityPub followers

* Minor adjustments
2017-08-13 00:44:41 +02:00
Eugen Rochko 5138dde794 Fix #4149, fix #1199 - Store emojis as unicode (#4189)
- Use unicode when selecting emoji through picker
- Convert shortcodes to unicode when storing text input server-side
- Do not convert shortcodes in JS anymore
2017-07-14 19:47:53 +02:00
Matt Jankowski 533a8cc68b Misc tidying and clean ups (#3445)
* Remove trailing whitespace in i18n mailers

* Use query methods instead of #present? on AR attributes

* Delegate Status#account_domain method

* Delegate Mention #account_username and #account_acct methods
2017-05-31 20:38:17 +02:00
Eugen Rochko 4a7dc4fadc OEmbed support for PreviewCard (#2337)
* OEmbed support for PreviewCard

* Improve ProviderDiscovery code failure treatment

* Do not crawl links if there is a content warning, since those
don't display a link card anyway

* Reset db schema

* Fresh migrate

* Fix rubocop style issues
Fix #1681 - return existing access token when applicable instead of creating new

* Fix test

* Extract http client to helper

* Improve oembed controller
2017-04-27 14:42:22 +02:00
Keiji, Yoshimi 8e7c858877 attach_media should be in transaction of creation status (#2426) 2017-04-26 03:47:44 +02:00
Eugen Rochko 553d6a1ea6 Fix #2402 - Add Idempotency-Key header to PostStatusService that prevents (#2419)
duplicates. Web UI regenerates UUID for that header every time the compose
form is changed or successfully submitted

Also, fix Farsi i18n overwriting the English one
2017-04-25 15:04:49 +02:00