Commit Graph

769 Commits (909b2257ab784f411ee68b210778826ee9d9106c)

Author SHA1 Message Date
ThibG 73051a4c10 Fix support for HTTP proxies (#11245)
* Disable incorrect check for hidden services in Socket

Hidden services can only be accessed with an HTTP proxy, in which
case the host seen by the Socket class will be the proxy, not the
target host.

Hidden services are already filtered in `Request#initialize`.

* Use our Socket class to connect to HTTP proxies

Avoid the timeout logic being bypassed

* Add support for IP addresses in Request::Socket

* Refactor a bit, no need to keep the DNS resolver around
2019-07-07 13:53:09 +02:00
Thibaut Girka d7a7f21646 Merge commit '64909cf0d988fb09917c41cfbc81b1e9a5132aca' into glitch-soc/merge-upstream
Unlike upstream, kept the direct timeline endpoint, as it is still of use in
glitch-soc.
2019-07-07 13:49:31 +02:00
Eugen Rochko ad6b03d4b6 Fix URLs appearing twice in errors of ActivityPub::DeliveryWorker (#11231) 2019-07-07 03:37:01 +02:00
ThibG ac1fec5be3 Fix support for HTTP proxies (#11245)
* Disable incorrect check for hidden services in Socket

Hidden services can only be accessed with an HTTP proxy, in which
case the host seen by the Socket class will be the proxy, not the
target host.

Hidden services are already filtered in `Request#initialize`.

* Use our Socket class to connect to HTTP proxies

Avoid the timeout logic being bypassed

* Add support for IP addresses in Request::Socket

* Refactor a bit, no need to keep the DNS resolver around
2019-07-07 02:05:38 +02:00
Eugen Rochko 4931208dd8 Remove Salmon and PubSubHubbub (#11205)
* Remove Salmon and PubSubHubbub endpoints

* Add error when trying to follow OStatus accounts

* Fix new accounts not being created in ResolveAccountService
2019-07-06 23:26:16 +02:00
ThibG 33607c43a5 Fix HTTP requests to IPv6 hosts (#11240) 2019-07-05 22:03:00 +02:00
Thibaut Girka 9866c2bfff Merge branch 'master' into glitch-soc/merge-upstream 2019-07-04 16:21:39 +02:00
Eugen Rochko 7391d223ab Fix statsd UDP sockets not being cleaned up in Sidekiq (#11230) 2019-07-02 01:01:17 +02:00
Eugen Rochko e782fcef4f Add request pool to improve delivery performance (#10353)
* Add request pool to improve delivery performance

Fix #7909

* Ensure connection is closed when exception interrupts execution

* Remove Timeout#timeout from socket connection

* Fix infinite retrial loop on HTTP::ConnectionError

* Close sockets on failure, reduce idle time to 90 seconds

* Add MAX_REQUEST_POOL_SIZE option to limit concurrent connections to the same server

* Use a shared pool size, 512 by default, to stay below open file limit

* Add some tests

* Add more tests

* Reduce MAX_IDLE_TIME from 90 to 30 seconds, reap every 30 seconds

* Use a shared pool that returns preferred connection but re-purposes other ones when needed

* Fix wrong connection being returned on subsequent calls within the same thread

* Reduce mutex calls on flushes from 2 to 1 and add test for reaping
2019-07-02 00:34:38 +02:00
Thibaut Girka ec1fcf1584 Use a redis-cached feed for the DM timeline 2019-06-27 16:44:12 +02:00
Thibaut Girka acbed64f3d Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/settings/preferences_controller.rb
- app/lib/user_settings_decorator.rb
- app/models/user.rb
- config/locales/simple_form.en.yml
2019-06-26 23:19:22 +02:00
ThibG 4b3d91dc2c Add option to disable blurhash previews (#11188)
* Add option to disable blurhash previews

* Update option text

* Change options order
2019-06-26 19:33:04 +02:00
ThibG 28806d5d6d Add support for Audio activities (#11189)
Fixes #11127
2019-06-26 19:32:36 +02:00
ThibG 42c803fe61 Apply filters to poll options (#11174)
* Apply filters to poll options in WebUI

Fixes #11128

* Apply filters to poll options server-side

* Add poll options to searchable text
2019-06-25 14:45:14 +02:00
Thibaut Girka 1177694c3e Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/media_attachment.rb
  Upstream added audio attachment support
- app/serializers/initial_state_serializer.rb
  Upstream added audio attachment support and how mimetypes are returned
- app/serializers/rest/instance_serializer.rb
  Upstream added a few fields
- config/application.rb
  Upstream added a different paperclip transcoder
2019-06-24 15:02:59 +02:00
Eugen Rochko d85b2a734b Fix audio not being downloaded from remote servers (#11145) 2019-06-22 02:50:36 +02:00
Eugen Rochko eac19f2300 Change domain blocks to automatically support subdomains (#11138)
* Change domain blocks to automatically support subdomains

If a more authoritative domain is blocked (example.com), then the
same block will be applied to a subdomain (foo.example.com)

* Match subdomains of existing accounts when blocking/unblocking domains

* Improve code style
2019-06-22 00:13:10 +02:00
Thibaut Girka b4b85301fc Fix rendering of emoji in public pages 2019-06-18 20:14:08 +02:00
Eugen Rochko ffbca47356 Fix sanitizer making block level elements unreadable (#10836)
Fix #10834
2019-06-16 21:46:36 +02:00
Thibaut Girka 5a158899bc Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/settings/notifications_controller.rb
- app/javascript/packs/public.js
- app/views/settings/preferences/show.html.haml
- app/views/stream_entries/_simple_status.html.haml
- config/locales/simple_form.en.yml
- config/locales/simple_form.pl.yml
- config/navigation.rb
- config/routes.rb
2019-06-10 18:59:53 +02:00
Eugen Rochko aed4fa02f1 Change priority of delete activity forwards for replies and reblogs (#11002)
Fix #11001
2019-06-09 12:47:33 +02:00
Thibaut Girka 00552d2f79 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/statuses_controller.rb
- app/controllers/stream_entries_controller.rb
2019-06-07 17:00:36 +02:00
ThibG be6e54a70c Do not misattribute inlined boosts if `attributedTo` isn't present (#10967)
* Do not misattribute inlined boosts if `attributedTo` isn't present

Fixes #10950

* Fix tests
2019-06-04 23:24:31 +02:00
Eugen Rochko 57098a1b8d Refactor all ActivityPub deliveries to be serialized and signed through one concern (#10966) 2019-06-04 23:11:18 +02:00
Thibaut Girka a76c660645 Merge remote-tracking branch 'upstream/master' into glitch-soc/merge-upstream 2019-06-04 20:40:19 +02:00
ThibG c45a330824 Handle blank poll options more gracefully (#10946)
Pleroma currently allows (erroneously imho) empty poll options, that is,
options with an empty (but existing) `name`.
2019-06-03 20:04:00 +02:00
Thibaut Girka 7fb3fa16dd Truncate long URLs while providing alt text for inline images 2019-05-28 19:33:43 +02:00
Thibaut Girka d417a32290 Translate incoming remote img tags by a link 2019-05-28 19:33:43 +02:00
Thibaut Girka d46668d34b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/settings/preferences_controller.rb
- app/lib/user_settings_decorator.rb
- app/models/user.rb

Conflicts due to the addition of a new preference upstream,
“advanced layout”.
2019-05-26 15:41:40 +02:00
Eugen Rochko a1bceb2cb6 Add responsive panels to the single-column layout (#10820)
* Add responsive panels to the single-column layout

* Fixes

* Fix not being able to save the preference

* Fix code style issues

* Set max-height on the compose textarea and add a link to relationship manager
2019-05-25 21:27:00 +02:00
ThibG 4e953ee471 Merge pull request #1067 from ThibG/glitch-soc/merge-upstream
Merge upstream changes
2019-05-23 22:01:34 +02:00
Thibaut Girka ce528d61b4 Do not share a single Redcarpet::Markdown object across threads
There are apparently thread-safety issues: https://github.com/vmg/redcarpet/pull/672
2019-05-23 21:19:44 +02:00
Thibaut Girka 630aafa3a7 Merge branch 'master' into glitch-soc/merge-upstream 2019-05-23 21:18:27 +02:00
ThibG 028fb2060b Fix possible race condition when processing statuses (#10815) 2019-05-23 20:00:39 +02:00
Thibaut Girka 3117d0ac70 Allow rel=tag in status text
Fixes tag links in local Markdown or HTML-authored statuses
2019-05-23 12:28:33 +02:00
Thibaut Girka a93c6f708b Fix local text/html toots not being sanitized 2019-05-21 23:12:43 +02:00
Thibaut Girka d48d1ed534 Minor optimization 2019-05-17 23:51:14 +02:00
Thibaut Girka 3d03d084cd Properly escape HTML in code blocks 2019-05-17 23:51:14 +02:00
Thibaut Girka 7aece1eec5 Add option for default toot content-type 2019-05-17 23:51:14 +02:00
Thibaut Girka ee3dfdf6a1 Fix autolinking, and newlines in code blocks
Autolinking is now performed *after* the Markdown pass, by replacing HTML
tags with zero-width spaces and running the twitter-text extractor as usual,
except it does not auto-link URLs to avoid links in links…
2019-05-17 23:51:14 +02:00
Thibaut Girka c9696ef8a5 Add support for HTML-formatted toots 2019-05-17 23:51:14 +02:00
Thibaut Girka 8313ee07f6 Add support for markdown-formatted toots 2019-05-17 23:51:14 +02:00
Thibaut Girka 422dd06806 Merge branch 'master' into glitch-soc/merge-upstream 2019-05-15 17:11:40 +02:00
ThibG 85a991cfa7 Minor performance improvements and cleanup in formatter (#10765) 2019-05-15 06:54:06 +02:00
Thibaut Girka 1bf48b01e3 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/statuses_controller.rb
  minor conflict because of glitch-soc's theming system
- app/controllers/stream_entries_controller.rb
  minor conflict because of glitch-soc's theming system
2019-05-10 17:09:12 +02:00
ThibG a571b07557 Prevent silenced local users from notifying remote users not following them (#10575)
* Prevent silenced local users from notifying remote users not following them

This is an attempt to extend the local restrictions of silenced users to the
federation.

* Add tests

* Add tests for making sure private status don't get sent over OStatus
2019-05-09 22:05:43 +02:00
Thibaut Girka 73bc58c656 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/media_attachment.rb
2019-05-04 16:37:26 +02:00
Thibaut Girka 81c9377c7d Add support for missing formatting tags 2019-04-29 19:31:09 +02:00
Eugen Rochko a6d2fe7165 Add blurhash (#10630)
* Add blurhash

* Use fallback color for spoiler when blurhash missing

* Federate the blurhash and accept it as long as it's at most 5x5

* Display unknown media attachments as blurhash placeholders

* Improve style of embed actions and spoiler button

* Change blurhash resolution from 3x3 to 4x4

* Improve dependency definitions

* Fix code style issues
2019-04-27 03:24:09 +02:00
Thibaut Girka 9aeb714b7a Add support for lists in statuses 2019-04-22 23:07:14 +02:00
Thibaut Girka 22ad8c89b2 Accept richer text from remote statuses
Support abbr, del, pre, blockquote, code, strong, b, em, i, and h1…h5
HTML elements in remote statuses, add corresponding CSS.
2019-04-22 23:07:14 +02:00
Thibaut Girka 6f85bb754b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/javascript/mastodon/features/compose/components/compose_form.js
  Upstream cleaned up a bit, including on lines in which
  we replaced the hardcoded 500 character limit with a maxChar
  constant. Applied the changes while keeping maxChar instead of 500.
- app/javascript/packs/public.js
  Moved upstream's new animated avatar hover handling in
  app/javascript/core/public.js
- app/javascript/styles/fonts/montserrat.scss
  Upstream fixed local font name, applied those changes.
- app/javascript/styles/fonts/roboto.scss
  Upstream fixed local font name, applied those changes.
- lib/mastodon/version.rb
  Upstream made repo URL configurable, did the same, but
  default to glitch-soc
2019-04-22 20:40:04 +02:00
Ben Lubar 483aa50856 Default to the web domain (eg. mastodon.lubar.me) instead of the local domain (eg. lubar.me) for keybase proofs (#10565) 2019-04-21 04:53:24 +02:00
Thibaut Girka 5a24934449 Merge branch 'master' into glitch-soc/merge-upstream 2019-04-10 21:19:21 +02:00
Eugen Rochko 6302bed0b5 Fix Keybase verification using wrong domain for remote accounts (#10547) 2019-04-10 20:28:43 +02:00
Thibaut Girka 5c8f9a3220 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- config/locales/pl.yml
  Conflict caused by new upstream string too close to glitch-specific
  “flavour” string. Took both strings.
2019-04-08 15:57:56 +02:00
Takeshi Umeda 21193ed31e Fix config serializer regex (#10487) 2019-04-08 07:55:38 +02:00
Thibaut Girka 99336afaff Merge branch 'master' into glitch-soc/merge-upstream 2019-03-30 12:36:24 +01:00
Eugen Rochko a82bc7f5ae Add ActivityPub representation for identity proofs (#10414)
* Add ActivityPub representation for identity proofs

* Add tests
2019-03-30 02:12:06 +01:00
Thibaut Girka 1600aeb6cb Merge branch 'master' into glitch-soc/merge-upstream 2019-03-28 18:35:25 +01:00
Alex Gessner be7c92061c squashed identity proof updates (#10375) 2019-03-28 18:01:09 +01:00
Thibaut Girka 0117de0a01 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/workers/activitypub/distribute_poll_update_worker.rb
- config/locales/pl.yml
2019-03-28 13:01:33 +01:00
Eugen Rochko 364e49ae1f Rename :poll to :preloadable_poll and :owned_poll to :poll on Status (#10401)
Also, fix some n+1 queries

Resolve #10365
2019-03-28 04:44:59 +01:00
Eugen Rochko 6e163d5b2a Fix alternative relay support regression (#10398)
Fix #10324
2019-03-27 19:58:24 +01:00
Eugen Rochko 367ab6764f Remove unused ActivityPub `@context` values depending on response (#10378)
Fix #8078
2019-03-27 15:55:23 +01:00
Thibaut Girka 1f70b51fff Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- config/locales/es.yml
- config/locales/pl.yml
- config/locales/pt-BR.yml
2019-03-26 16:02:54 +01:00
Eugen Rochko 0e730974cd Change language detector threshold from 140 characters to 4 words (#10376)
Add `lang` attribute to statuses in web UI
2019-03-26 01:23:59 +01:00
Thibaut Girka 8410fcd3a7 Merge branch 'master' into glitch-soc/merge-upstream 2019-03-22 13:05:17 +01:00
Eugen Rochko d20386e209 Improve config serializer for Keybase (#10338)
- Regex must no longer be surrounded by `/`
- Description must be short and cannot contain HTML tags
2019-03-21 23:33:28 +01:00
Thibaut Girka f030541264 Merge branch 'master' into glitch-soc/merge-upstream 2019-03-20 17:32:39 +01:00
ThibG 2a211ff18c Add support for custom emojis in poll options (#10322)
* Backend changes for custom emoji support in poll options

* Serialize poll emojis in REST API

* Render custom emojis in poll options

* Render custom emoji in poll options on public pages
2019-03-20 17:29:12 +01:00
Thibaut Girka c37bc0b268 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- config/locales/en.yml
  Conflict caused by the glitch-soc-specific “flavour” string being too close
  to the newly introduced “identity_proofs” string. Just included both.
2019-03-20 13:54:00 +01:00
Eugen Rochko a3c41f3532 Add Keybase integration (#10297)
* create account_identity_proofs table

* add endpoint for keybase to check local proofs

* add async task to update validity and liveness of proofs from keybase

* first pass keybase proof CRUD

* second pass keybase proof creation

* clean up proof list and add badges

* add avatar url to keybase api

* Always highlight the “Identity Proofs” navigation item when interacting with proofs.

* Update translations.

* Add profile URL.

* Reorder proofs.

* Add proofs to bio.

* Update settings/identity_proofs front-end.

* Use `link_to`.

* Only encode query params if they exist.

URLs without params had a trailing `?`.

* Only show live proofs.

* change valid to active in proof list and update liveness before displaying

* minor fixes

* add keybase config at well-known path

* extremely naive feature flagging off the identity proof UI

* fixes for rubocop

* make identity proofs page resilient to potential keybase issues

* normalize i18n

* tweaks for brakeman

* remove two unused translations

* cleanup and add more localizations

* make keybase_contacts an admin setting

* fix ExternalProofService my_domain

* use Addressable::URI in identity proofs

* use active model serializer for keybase proof config

* more cleanup of keybase proof config

* rename proof is_valid and is_live to proof_valid and proof_live

* cleanup

* assorted tweaks for more robust communication with keybase

* Clean up

* Small fixes

* Display verified identity identically to verified links

* Clean up unused CSS

* Add caching for Keybase avatar URLs

* Remove keybase_contacts setting
2019-03-18 21:00:55 +01:00
Thibaut Girka 365de06816 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/accounts_controller.rb
- app/controllers/follower_accounts_controller.rb
- app/controllers/statuses_controller.rb

All conflicts caused by the additional `use_pack` used for glitch-soc's theming
system.
2019-03-18 18:03:27 +01:00
ThibG 9efb9d7f72 Set and store report URIs (#10303)
Fixes #10271
2019-03-17 15:34:56 +01:00
Thibaut Girka eed61a37ec Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/settings/follower_domains_controller.rb
  Removed upstream. Did the same here. Maybe we should not have?
- config/locales/en.yml
  Upstream removed the “Authorized followers” page and associated
  translations. This is too close in the file to our glitch-soc-specific
  “flavour” string. No actual conflict.
- config/locales/ja.yml
  Same as above.
- config/locales/pl.yml
  Same as above.
- config/navigation.rb
  No real conflict. New route added too close to the glitch-soc-specific
  “flavours” one.
- config/webpack/configuration.js
  Upstream refactored the webpack(er) configuration quite a bit.
  Tried to keep up.
- config/webpack/loaders/babel.js
  Upstream refactored the webpack(er) configuration quite a bit.
  Tried to keep up.
  The contents of this file have been moved to package.json.
- config/webpack/shared.js
  Upstream refactored the webpack(er) configuration quite a bit.
  Tried to keep up.
- config/webpacker.yml
  Upstream refactored the webpack(er) configuration quite a bit.
  Tried to keep up.
- jest.config.js
  The contents of this file have been moved to package.json.
- package.json
  Upstream refactored the webpack(er) configuration quite a bit.
  Tried to keep up.
- yarn.lock
  Upstream refactored the webpack(er) configuration quite a bit.
  Tried to keep up.
2019-03-16 14:14:20 +01:00
Eugen Rochko 793587060b Fix language detection of non-latin alphabets even at few characters (#10276) 2019-03-15 05:07:09 +01:00
Thibaut Girka d86033c352 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/about_controller.rb
- app/controllers/tags_controller.rb
- app/views/about/show.html.haml
- spec/views/about/show.html.haml_spec.rb
2019-03-13 15:16:02 +01:00
Eugen Rochko d5377d0d45 Fix poll update handler calling method was that was not available (#10246)
* Fix poll update handler calling method was that was not available

Fix regression from #10209

* Refactor VoteService

* Refactor ActivityPub::DistributePollUpdateWorker and optimize it

* Fix typo

* Fix typo
2019-03-12 22:58:59 +01:00
Thibaut Girka edd48921c6 Merge branch 'master' into glitch-soc/merge-upstream 2019-03-11 17:31:02 +01:00
ThibG 154c8a0302 Increase DNS timeouts (#10238) 2019-03-11 13:27:57 +01:00
Thibaut Girka 33001891b6 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/services/post_status_service.rb
  Small conflict due to handling of instance-local toots.
  A subsequent change is required to ensure instance-local polls are not leaked
  through Update.
2019-03-11 11:23:50 +01:00
ThibG 11ce444b8b Support pushing and receiving updates to poll tallies (#10209)
* Process incoming poll tallies update

* Send Update on poll vote

* Do not send Updates for a poll more often than once every 3 minutes

* Include voters in people to notify of results update

* Schedule closing poll worker on poll creation

* Add new notification type for ending polls

* Add front-end support for ended poll notifications

* Fix UpdatePollSerializer

* Fix Updates not being triggered by local votes

* Fix tests failure

* Fix web push notifications for closing polls

* Minor cleanup

* Notify voters of both remote and local polls when those close

* Fix delivery of poll updates to mentioned accounts and voters
2019-03-11 00:49:31 +01:00
Thibaut Girka 9b093dfcde Merge branch 'master' into glitch-soc/merge-upstream 2019-03-08 00:57:13 +01:00
ThibG dcbe6c16ba Do not allow adding votes to expired polls (#10214)
* Do not allow adding votes to expired polls

* Only validate expires_at on create
2019-03-08 00:54:50 +01:00
Thibaut Girka d8d8c64965 Merge branch 'master' into glitch-soc/merge-upstream 2019-03-07 13:24:30 +01:00
ThibG db21a6c5ee Avoid unnecessarily fetching the replies collection when it is empty (#10201) 2019-03-07 01:50:37 +01:00
Thibaut Girka 9cb4a4c14e Merge branch 'master' into glitch-soc/merge-upstream 2019-03-06 00:03:46 +01:00
ThibG 46f837a107 Fix newlines in OStatus and RSS serializations (#10183) 2019-03-05 23:58:58 +01:00
Thibaut Girka 4c06fafeae Merge branch 'master' into glitch-soc/merge-upstream 2019-03-05 21:40:28 +01:00
ThibG acbf18b517 When serializing polls over OStatus, serialize poll options to text (#10160)
* When serializing polls over OStatus, serialize poll options to text

* Do the same for RSS feeds

* Use “[ ] ” as a prefix for poll options instead of “- ”
2019-03-05 21:09:18 +01:00
Thibaut Girka c0b876f726 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/status.rb
- db/schema.rb

Both conflicts are caused by us having extra database columns.
2019-03-05 19:23:16 +01:00
ThibG 7da087dabe Store remote votes URI (#10158)
* Store remote votes URI

* Add spec for accepting remote votes

* Make poll vote id generation work the same way as follows
2019-03-04 22:51:23 +01:00
Eugen Rochko 03b5da7b6f Add tests for ActivityPub poll processing (#10143) 2019-03-04 01:13:42 +01:00
Eugen Rochko 952045604c Fix remote poll expiration time (#10144) 2019-03-04 00:44:34 +01:00
Eugen Rochko 4a3acdc916 Add polls (#10111)
* Add polls

Fix #1629

* Add tests

* Fixes

* Change API for creating polls

* Use name instead of content for votes

* Remove poll validation for remote polls

* Add polls to public pages

* When updating the poll, update options just in case they were changed

* Fix public pages showing both poll and other media
2019-03-03 22:18:23 +01:00
Thibaut Girka 91934eeb74 Merge branch 'master' into glitch-soc/merge-upstream 2019-02-28 21:35:53 +01:00
ThibG 892327c686 Give the `replies` collection an identifier and enable pagination (#10128) 2019-02-28 18:16:34 +01:00
ThibG 9f3f47e91f Improved remote thread fetching (#10106)
* Fetch up to 5 replies when discovering a new remote status

This is used for resolving threads downwards. The originating
server must add a “replies” attributes with such replies for it to
be useful.

* Add some tests for ActivityPub::FetchRepliesWorker

* Add specs for ActivityPub::FetchRepliesService

* Serialize up to 5 public self-replies for ActivityPub notes

* Add specs for ActivityPub::NoteSerializer

* Move exponential backoff logic to a worker concern

* Fetch first page of paginated collections when fetching thread replies

* Add specs for paginated collections in replies

* Move Note replies serialization to a first CollectionPage

The collection isn't actually paginable yet as it has no id nor
a `next` field. This may come in another PR.

* Use pluck(:uri) instead of map(&:uri) to improve performances

* Fix fetching replies when they are in a CollectionPage
2019-02-28 15:22:21 +01:00
ThibG 181c40a402 Fix mention processing for unknwon accounts on incoming ActivityPub Notes (#10125)
`::FetchRemoteAccountService` is not `ActivityPub::FetchRemoteAccountService`,
its second argument is the pre-fetched body. Passing `id: false` actually passed
a `Hash` as the prefetched body, instead of properly resolving unknown remote
accounts.
2019-02-27 14:57:14 +01:00
Thibaut Girka 561520a2be Merge branch 'master' into glitch-soc/merge-upstream 2019-02-17 15:43:12 +01:00
Eugen Rochko 6068b479f6 Fix Announce activities of unknown statuses not fetching those statuses (#10065)
Regression from #9998
2019-02-17 15:16:36 +01:00
Thibaut Girka da24c6aced Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/serializers/rest/instance_serializer.rb
- app/views/layouts/error.html.haml
- public/oops.png
  Took our version (same as upstream but without the things that only
  make sense in an animation).

Additional changes:
- app/javascript/flavours/vanilla/theme.yml
  Include upstream's javascript in error pages.
2019-02-17 12:26:55 +01:00
Eugen Rochko 397f180493 Add logging for rejected ActivityPub payloads and add tests (#10062) 2019-02-17 03:38:25 +01:00
Eugen Rochko 0230b3c41d Filter incoming Announce activities by relation to local activity (#10041)
* Filter incoming Announce activities by relation to local activity

Reject if announcer is not followed by local accounts, and is not
from an enabled relay, and the object is not a local status

Follow-up to #10005

* Fix tests
2019-02-15 18:19:45 +01:00
Thibaut Girka c8086b6efb Merge branch 'master' into glitch-soc/merge-upstream 2019-02-15 18:02:45 +01:00
Eugen Rochko ddbf75ea87 Filter incoming Create activities by relation to local activity (#10005)
Reject those from accounts with no local followers, from relays
that are not enabled, which do not address local accounts and are
not replies to accounts that do have local followers
2019-02-13 18:42:47 +01:00
ThibG 99dcb6d910 Alternative handling of private self-boosts (#9998)
* When self-boosting, embed original toot into Announce serialization

* Process unknown self-boosts from Announce object if it is more than an URI

* Add some self-boost specs

* Only serialize private toots in self-Announces
2019-02-13 18:36:23 +01:00
Thibaut Girka 1c37ad108c Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/oauth/authorized_applications_controller.rb
  Two changes too close to each other
- app/controllers/settings/sessions_controller.rb
- app/lib/user_settings_decorator.rb
  Two changes too close to each other
- app/models/media_attachment.rb
  New changes too close to glitch-soc only changes.
- app/models/user.rb
  Two changes too close to each other.
- app/services/remove_status_service.rb
  Kept direct timeline code which had been removed upstream.
- app/views/settings/preferences/show.html.haml
  Two changes too close to each other.
- config/locales/en.yml
  Introduction of a new string too close to glitch-soc-only's “flavour”
- config/locales/ja.yml
  Introduction of a new string too close to glitch-soc-only's “flavour”
- config/locales/pl.yml
  Introduction of a new string too close to glitch-soc-only's “flavour”
- config/locales/simple_form.en.yml
  Introduction of a new string too close to glitch-soc-only's “skin”
- config/locales/simple_form.pl.yml
  Introduction of a new string too close to glitch-soc-only's “skin”
- config/settings.yml
  Reverted upstream's decision of enabling posting application by default.
2019-02-10 21:10:09 +01:00
Eugen Rochko 5adc3f5676 Fix URL linkifier grabbing full-width spaces and quotations (#9997)
Fix #9993
Fix #5654
2019-02-09 20:13:11 +01:00
Hinaloe cce3319bc2 Only URLs extract with pre-escaped text (#9991)
* [test] add japanese hashtag testcase

* Only URLs extract with pre-escaped text

( https://github.com/tootsuite/mastodon/issues/9989 )
2019-02-09 03:39:38 +01:00
ThibG dfb101cd45 Make displaying application used to toot opt-in (#9897)
* Make storing and displaying application used to toot opt-in

* Revert to storing application info, and display it to the author via API
2019-02-02 19:18:15 +01:00
ysksn 9cad360caa Create Redisable#redis (#9633)
* Create Redisable

* Use #redis instead of Redis.current
2019-02-02 19:11:38 +01:00
Jakub Mendyk 234e8105da Allow most kinds of characters in URL query (fixes #8408) (#8447)
* Allow unicode characters in URL query strings

Fixes #8408

* Alternative approach to unicode support in urls

Adds PoC/idea to approch this problem.
2019-02-02 19:01:18 +01:00
Thibaut Girka bb035bb0ab Make storing and displaying application used to toot opt-in 2019-01-27 13:56:41 +01:00
Thibaut Girka b49d415f37 Merge branch 'master' into glitch-soc/merge-upstream
No conflicts.
2019-01-19 18:28:37 +01:00
ThibG d0008b5051 Add tombstones for remote statuses (#9830)
* Add Tombstone model to remember object deletion

* Do not recreate a status if it has been deleted

* Record Tombstone for remote deleted items

Also, only record deleted items from same-host actors

* Clear an user's tombstones when their key change
2019-01-18 15:56:55 +01:00
Eugen Rochko 77f972d5b1 Add support for non-public reblogs from ActivityPub (#9841)
Fix #9838
2019-01-18 15:56:21 +01:00
Eugen Rochko 9b0c7cfa6f Use summary as summary for converted ActivityPub objects (#9823)
Fix #8609
2019-01-16 18:36:17 +01:00
ThibG b8867a3412 Reduce chances of race conditions when processing deleted toots (#9815)
* Reduce chances of race conditions when processing deleted toots

* Prevent race condition when processing deleted toots
2019-01-16 15:42:00 +01:00
Thibaut Girka 8d47495251 Revert "Revert "Add handler for Move activity (#9629)""
This reverts commit 6ab727bf6a.
2019-01-10 18:46:17 +01:00
Thibaut Girka 6ab727bf6a Revert "Add handler for Move activity (#9629)"
This reverts commit 4e3131f02f.
2019-01-02 15:36:59 +01:00
Thibaut Girka 44f2224606 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts manually resolved:
- app/services/post_status_service.rb
- config/locales/simple_form.pl.yml
- config/routes.rb
- config/webpack/loaders/sass.js
- config/webpack/shared.js
- package.json
- yarn.lock
2019-01-02 15:36:53 +01:00
ThibG ffcaa4719a Ensure blocked user unfollows blocker if Block/Undo Block are processed out of order (#9687)
* Ensure blocked user unfollows blocker if Block/Undo Block are processed out of order

* Add specs for Block causing unfollow and for out-of-order Block + Undo
2019-01-02 01:12:02 +01:00
ThibG 9bc77287d2 Do not ignore federated reports targetting already-reported accounts (#9534) 2018-12-30 18:58:51 +01:00
ThibG bdc44c3558 Reduce usage of LD signatures (#9659)
* Do not LDS-sign Follow, Accept, Reject, Undo, Block

* Do not use LDS for Create activities of private toots

* Minor cleanup

* Ignore unsigned activities instead of misattributing them

* Use status.distributable? instead of querying visibility directly
2018-12-30 09:48:59 +01:00
Eugen Rochko 4e3131f02f Add handler for Move activity (#9629) 2018-12-29 02:24:36 +01:00
Eugen Rochko ab9308b7f7 Fix ThreadResolveWorker getting queued with invalid URLs (#9628) 2018-12-26 19:15:53 +01:00
Thibaut Girka cabf2772ea Add a per-user setting to hide followers count
This is only available if the instance-wide setting isn't set and allows
people to hide their own followers count. This does not hide others' to
them.
2018-12-20 14:51:12 +01:00
Thibaut Girka b62953b3d2 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/javascript/packs/public.js
- app/models/user.rb
- config/settings.yml
- db/schema.rb

Moved public.js changes to settings.js.
2018-12-09 16:08:04 +01:00
ThibG 097d866028 Add setting to not aggregate reblogs (#9248)
* Add setting to not aggregate reblogs

Fixes #9222

* Handle cases where user is nil in add_to_home and add_to_list

* Add hint for setting_aggregate_reblogs option

* Reword setting_aggregate_reblogs label
2018-12-09 13:03:01 +01:00
Thibaut Girka 77b9249250 Merge branch 'master' into glitch-soc/merge-upstream 2018-11-29 19:22:52 +01:00
Thibaut Girka b2e9a23ac9 Add database support for list show-reply preferences 2018-11-28 14:46:07 +01:00
Eugen Rochko 030ce53623 Fix TLS handshake timeout not being enforced (#9381)
Follow-up to #9329
2018-11-27 19:46:05 +01:00
Eugen Rochko 8c7ba82dc1 Fix nil error when no DNS addresses are found for host (#9379) 2018-11-27 18:13:36 +01:00
Thibaut Girka 478abe3a58 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/status.rb

Resolved by taking both changes (not a real conflict, just changes too close
to each other).
2018-11-27 13:23:02 +01:00
Eugen Rochko f3fee3cd27 Fix connect timeout not being enforced (#9329)
* Fix connect timeout not being enforced

The loop was catching the timeout exception that should stop execution, so the next IP would no longer be within a timed block, which led to requests taking much longer than 10 seconds.

* Use timeout on each IP attempt, but limit to 2 attempts

* Fix code style issue

* Do not break Request#perform if no block given

* Update method stub in spec for Request

* Move timeout inside the begin/rescue block

* Use Resolv::DNS with timeout of 1 to get IP addresses

* Update Request spec to stub Resolv::DNS instead of Addrinfo

* Fix Resolve::DNS stubs in Request spec
2018-11-22 20:12:04 +01:00
ThibG 6d38e81021 Include replies to list owner and replies to list members in list statuses (#9324) 2018-11-21 17:02:58 +01:00
Eugen Rochko 0ebb6c1480 Revert connect timeout from 1s to 10s (#9319)
The failure rate in Sidekiq is too high
2018-11-21 17:00:56 +01:00
Eugen Rochko 4ec133ef93 Prevent multiple handlers for Delete of Actor from running (#9292) 2018-11-16 19:46:23 +01:00
Eugen Rochko 7a939f7cfc Remove intermediary arrays when creating hash maps from results (#9291) 2018-11-16 15:02:18 +01:00
Thibaut Girka b3553f992d Merge branch 'master' into glitch-soc/merge-upstream 2018-11-13 16:28:07 +01:00
ThibG b6e377a340 Fix emoji update date processing (#9255) 2018-11-10 23:59:51 +01:00
Thibaut Girka bf26c23205 Merge branch 'master' into glitch-soc/merge-upstream 2018-11-09 14:56:31 +01:00
Eugen Rochko d78aed7a37 Reduce connect timeout limit and limit signature failures by source IP (#9236)
* Reduce connect timeout from 10s to 1s

* Limit failing signature verifications per source IP
2018-11-08 21:35:58 +01:00
Thibaut Girka fd18ed4cea Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/auth/sessions_controller.rb

Upstream reverted something we partially reverted already.
Reverted the rest to match upstream.
2018-10-30 17:52:08 +01:00
Eugen Rochko d84886f35e Accept the same payload in multiple inboxes and deliver (#9150) 2018-10-30 15:03:55 +01:00
Thibaut Girka 13ea29754d Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- config/locales/simple_form.ja.yml

Not a real conflict, one new string was introduced in
glitch and one in upstream.
2018-10-29 14:00:41 +01:00
m.b a342451341 Add Page AP type support (#9121) 2018-10-29 13:23:29 +01:00
Thibaut Girka 65cc5faf80 Merge branch 'master' into glitch-soc/merge-upstream 2018-10-27 18:54:26 +02:00