Commit Graph

726 Commits (81acdc4151fc0df64e7b77a53d8d9ae8cc2faff3)

Author SHA1 Message Date
nightpool d4ddfc486a microformat mentions can have an implicit property (#12189)
See the first example here: http://microformats.org/wiki/microformats2#hyperlinked_person
2019-10-24 22:46:15 +02:00
Eugen Rochko b665e99b2f Fix hashtag timeline REST API accepting too many hashtags (#12091) 2019-10-07 03:56:21 +02:00
Eugen Rochko 059287ad08 Add `reason` param to `POST /api/v1/accounts` REST API (#12064)
For approval-required registrations mode
2019-10-03 17:50:59 +02:00
ThibG 081cae4c22 Fix BootstrapTimelineService crashing when bootstrapped accounts are invalid (#12037)
* Add test to handle suspended and missing users in BootstrapTimelineService

* Fix BootstrapTimelineService crashing when bootstrapped accounts are invalid
2019-10-01 15:10:00 +02:00
Eugen Rochko 67713f8b08 Remove HEAD request from fetching link previews (#12028)
It is not really necessary and we need to reduce requests
2019-10-01 04:54:10 +02:00
abcang 2b89873dfc Fix follow requests N+1 (#12004) 2019-09-30 17:02:03 +02:00
ThibG f088e090e4 Add voters count support (#11917)
* Add voters count to polls

* Add ActivityPub serialization and parsing of voters count

* Add support for voters count in WebUI

* Move incrementation of voters count out of redis lock

* Reword “voters” to “people”
2019-09-29 22:58:01 +02:00
Eugen Rochko ae03161ad9 Fix account migration not affecting followers on origin server (#11980) 2019-09-29 16:23:13 +02:00
Eugen Rochko 386dc65671 Fix preview card image not being re-fetched even if link is re-posted (#11981)
Fix #11956
2019-09-28 01:33:16 +02:00
Eugen Rochko 66d3e13777 Add `exclude_unreviewed` param to `GET /api/v2/search` REST API (#11977)
Make it so normal search returns even unreviewed matches, but
autosuggestions do not.

Fix #11960
2019-09-28 01:02:21 +02:00
ThibG 70990720c5 Change silences to always require approval on follow (#11975)
* Change silenced accounts to require approval on follow

* Also require approval for follows by people explicitly muted by target accounts

* Do not auto-accept silenced or muted accounts when switching from locked to unlocked

* Add `follow_requests_count` to verify_credentials

* Show “Follow requests” menu item if needed even if account is locked

* Add tests

* Correctly reflect that follow requests weren't auto-accepted when local account is silenced

* Accept follow requests from user-muted accounts to avoid leaking mutes
2019-09-27 21:13:51 +02:00
Eugen Rochko e2372fcdc2 Fix unreviewed hashtag not being found by exact case-insensitive match (#11976) 2019-09-27 19:40:26 +02:00
Jeong Arm 565a36ab61 Fix search error when ElasticSearch is enabled but not available (#11954)
* Fallback to Database search when ES not available

* Prevent double work if ES gives 0 result

* Apply suggestion from code review
2019-09-26 18:06:08 +02:00
Eugen Rochko 195ea5bce5 Change spam check to apply to local accounts and add a threshold (#11806)
Instead of detecting spam on first duplicate message, add a
threshold of 5 such messages to reduce false positives
2019-09-18 12:53:13 +02:00
Eugen Rochko d321515c11 Fix Move handler queuing jobs that will fail if account is suspended (#11864)
Don't put Move handler on cooldown if it didn't run. Skip unmerging
from timelines to save unnecessary work.
2019-09-17 08:44:45 +02:00
Eugen Rochko ee1c152083 Fix search API not resolving URL when type is given (#11822)
Fix #11771
2019-09-13 16:01:39 +02:00
Eugen Rochko 4291b74031 Change deletes to preserve soft-deleted statuses in unresolved reports (#11805)
Change all account actions except "none" to resolve all unresolved reports

Refactor `SuspendAccountService` to be more readable
2019-09-11 16:32:44 +02:00
Eugen Rochko 7ef93513f7 Change account deletion page to have better explanations (#11753)
Fix deletion of unconfirmed account not freeing up the username

Add prefill of logged-in user's email in the reconfirmation form
2019-09-04 04:13:54 +02:00
Eugen Rochko 79922ae20f Add profile directory to web UI (#11688)
* Add profile directory to web UI

* Add a line of bio to the directory
2019-08-30 00:14:36 +02:00
Eugen Rochko e4605bbb4a Fix remote and staff-removed statuses leaving media behind for a day (#11638)
The reason for unattaching media instead of removing it is to support
delete & redraft functionality, but remote or staff-removed statuses
will never be redrafted, so the media should be deleted immediately
2019-08-22 04:17:12 +02:00
Eugen Rochko 848cb2a697 Add more accurate hashtag search (#11579)
* Add more accurate hashtag search

Using ElasticSearch to index hashtags with edge n-grams and score
them by usage within the last 7 days since last activity. Only
hashtags that have been reviewed and are listable can appear in
searches, unless they match the query exactly

* Fix search analyzer dropping non-ascii characters
2019-08-18 03:45:51 +02:00
Eugen Rochko 291f316022 Fix accounts search by full/partial display name and others (#11580)
- Restrict followers counts to local users to minimize local advantage
- Fix emoji shortcodes causing error in search
- Fix search syntax parse errors not being caught
2019-08-16 13:00:30 +02:00
Eugen Rochko 6393a094c6 Add more accurate account search (#11537)
* Add more accurate account search

When ElasticSearch is available, a more accurate search is implemented:

- Using edge n-gram index for acct and display name
- Using asciifolding and cjk width normalization on display names
- Using Gaussian decay on account activity for additional scoring (recency)
- Using followers/friends ratio for additional scoring (spamminess)
- Using followers number for additional scoring (size)

The exact match precedence only takes effect when the input conforms
to the username format and the username part of it is complete, i.e.
when the user started typing the domain part.

* Support single-letter usernames

* Fix tests

* Fix not picking up account updates

* Add weights and normalization for scores, skip zero terms queries

* Use local counts for accounts index, adjust search parameters

* Fix mistakes

* Using updated_at of accounts is inadequate for remote accounts
2019-08-16 01:24:03 +02:00
Eugen Rochko 826bc61c2c Fix account search always returning exact match on paginated results (#11525)
Fix #11365
2019-08-11 13:14:51 +02:00
Eugen Rochko 10b486f627 Fix acct URIs with IDN domains not being resolved (#11520)
Fix #11494
2019-08-07 21:14:08 +02:00
ThibG 072e696aa1 Add domain block notes (#11515)
* Add database columns for adding notes to domain blocks/restrctions

* Add admin UI to set private and public comments when blocking a domain

* Add text for private and public comments on domain blocks

* Show domain block comments in admin UI

* Add comments to the domain block undo page

* Make UnblockDomainService more robust regarding upgraded domain blocks

* Allow editing domain blocks

* Rename button from “undo domain block” to “view domain block” in account admin UI

* Change test to unsilence silenced users from upgraded blocks
2019-08-07 20:20:23 +02:00
Eugen Rochko 277acb8772 Fix non-lowercase hashtags not being picked up by the streaming API (#11508)
Regression from b838607

Fix hashtag links always being lowercase
2019-08-07 10:01:19 +02:00
Eugen Rochko 825dc3ca22 Add whitelist mode (#11291) 2019-07-30 11:10:46 +02:00
Eugen Rochko b8386076c5 Change hashtags to preserve first-used casing (#11416) 2019-07-28 05:59:51 +02:00
Eugen Rochko 9c617b3b3a Add search syntax for operators and phrases (#11411) 2019-07-27 04:42:08 +02:00
ThibG 52cda33b82 Fix invites not being disabled upon account suspension (#11412)
* Disable invite links from disabled/suspended users

* Add has_many invites relationship to users

* Destroy unused invites when suspending an account
2019-07-26 18:55:33 +02:00
Eugen Rochko 93a6ab5a32 Change account domain block to clear out notifications and follows (#11393) 2019-07-25 04:17:35 +02:00
Eugen Rochko f9ed4354b1 Remove WebSub subscriptions (#11303) 2019-07-21 04:08:00 +02:00
ThibG a7b0224699 Disable LDSigning when AUTHORIZED_FETCH is set to true (#11295) 2019-07-15 02:29:39 +02:00
ThibG 3f12a0b8fd Fix leaking private statuses the admin account follows (#11300)
Now that the request is signed, it can return private toots. Do not leak them.
2019-07-15 02:29:04 +02:00
Eugen Rochko f70b20a01c Add a spam check (#11217)
* Add a spam check

* Use Nilsimsa to generate locality-sensitive hashes and compare using Levenshtein distance

* Add more tests

* Add exemption when the message is a reply to something that mentions the sender

* Use Nilsimsa Compare Value instead of Levenshtein distance

* Use MD5 for messages shorter than 10 characters

* Add message to automated report, do not add non-public statuses to
automated report, add trust level to accounts and make unsilencing
raise the trust level to prevent repeated spam checks on that account

* Expire spam check data after 3 months

* Add support for local statuses, reduce expiration to 1 week, always create a report

* Add content warnings to the spam check and exempt empty statuses

* Change Nilsimsa threshold to 95 and make sure removed statuses are removed from the spam check

* Add all matched statuses into automatic report
2019-07-13 16:45:50 +02:00
ThibG 4bf0ee9467 Fix BlockService trying to reject incorrect follow request (#11288)
Fixes #11148
2019-07-11 14:50:27 +02:00
Eugen Rochko 317b79d673 Add HTTP signatures to all outgoing ActivityPub GET requests (#11284) 2019-07-11 14:49:55 +02:00
Eugen Rochko 6baf5099a6 Refactor fetching of remote resources (#11251) 2019-07-10 18:59:28 +02:00
ThibG e39e4786b4 Fix handling of webfinger redirects in ResolveAccountService (#11279) 2019-07-10 17:10:12 +02:00
Eugen Rochko d0b0b63b1a Refactor domain block checks (#11268) 2019-07-09 03:27:35 +02:00
Eugen Rochko 56f0203c66 Refactor controllers for statuses, accounts, and more (#11249) 2019-07-08 12:03:45 +02:00
ThibG 12d6c0ff9b Fix BackupService crashing when an attachment is missing (#11241)
* Fix BackupService crashing when an attachment is missing

For various reasons such as admin error or out-of-sync media and
database backups, it might be possible for local attachments to be lost.

This commit allows the BackupService to continue its work even if some media
file is missing.

* Change error message
2019-07-08 02:24:25 +02:00
Eugen Rochko fbbcbd940d Remove Atom feeds and old URLs in the form of `GET /:username/updates/:id` (#11247) 2019-07-07 16:16:51 +02:00
Eugen Rochko 4931208dd8 Remove Salmon and PubSubHubbub (#11205)
* Remove Salmon and PubSubHubbub endpoints

* Add error when trying to follow OStatus accounts

* Fix new accounts not being created in ResolveAccountService
2019-07-06 23:26:16 +02:00
Eugen Rochko 9ac82421f8 Change domain block behaviour to prevent creation of accounts from suspended domains (#11219) 2019-07-01 19:13:30 +02:00
ThibG 28806d5d6d Add support for Audio activities (#11189)
Fixes #11127
2019-06-26 19:32:36 +02:00
Eugen Rochko eac19f2300 Change domain blocks to automatically support subdomains (#11138)
* Change domain blocks to automatically support subdomains

If a more authoritative domain is blocked (example.com), then the
same block will be applied to a subdomain (foo.example.com)

* Match subdomains of existing accounts when blocking/unblocking domains

* Improve code style
2019-06-22 00:13:10 +02:00
Eugen Rochko d7e2f554fb Add audio uploads (#11123)
* Add audio uploads

Fix #4827

Accept uploads of OGG, WAV, FLAC, OPUS and MP3 files, and converts
them to OGG. Media attachments get a new `audio` type. In the UI,
audio uploads are displayed identically to video uploads.

* Improve code style
2019-06-19 23:42:38 +02:00
Eugen Rochko 57098a1b8d Refactor all ActivityPub deliveries to be serialized and signed through one concern (#10966) 2019-06-04 23:11:18 +02:00