Commit Graph

1171 Commits (c5925756f13398f42fb1a59f8504050d195888a8)

Author SHA1 Message Date
Thibaut Girka 050efbc126 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/workers/activitypub/distribute_poll_update_worker.rb
- config/locales/pl.yml
2019-03-28 13:01:33 +01:00
Eugen Rochko f1bc90ab50
Rename :poll to :preloadable_poll and :owned_poll to :poll on Status (#10401)
Also, fix some n+1 queries

Resolve #10365
2019-03-28 04:44:59 +01:00
Thibaut Girka fcce135d27 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/admin/settings_controller.rb
- app/models/form/admin_settings.rb

Conflicts caused by upstream refactoring, while we have
flavours and skins, with the flavour_and_skin pseudo-setting.
2019-03-26 16:01:59 +01:00
Eugen Rochko 9d0573f574
Fix vote-after-expiration not showing error, count vote as interaction (#10352) 2019-03-23 14:07:17 +01:00
Thibaut Girka edd5441112 Merge branch 'master' into glitch-soc/merge-upstream 2019-03-22 13:05:17 +01:00
ThibG 62dd0b7f9c Fix processing poll Updates (#10333)
ActivityPub::ProcessPollService was checking the JSON-LD context although
it was passed only the `Question` object embedded in the `Update` activity.
2019-03-21 14:24:09 +01:00
Thibaut Girka 1d6152f440 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- config/locales/en.yml
  Conflict caused by the glitch-soc-specific “flavour” string being too close
  to the newly introduced “identity_proofs” string. Just included both.
2019-03-20 13:54:00 +01:00
ThibG 1217185cfa Do not distribute Delete when rejecting unapproved accounts (#10321) 2019-03-19 16:33:30 +01:00
Thibaut Girka 291012ec0c Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/accounts_controller.rb
- app/controllers/follower_accounts_controller.rb
- app/controllers/statuses_controller.rb

All conflicts caused by the additional `use_pack` used for glitch-soc's theming
system.
2019-03-18 18:03:27 +01:00
ThibG a20354a20b Set and store report URIs (#10303)
Fixes #10271
2019-03-17 15:34:56 +01:00
Thibaut Girka 4aa6aba6ee Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/services/remove_status_service.rb
  Conflict because we still handle the direct timeline.
  Took upstream changes, with that one extra function call.
- config/locales/ca.yml
  Conflict because theme names were updated.
  Decided to *keep* the theme names even if they are useless
  to avoid future conflicts.
- config/locales/oc.yml
  Decided to *keep* the theme names even if they are useless
  to avoid future conflicts.
- config/locales/pl.yml
  Decided to *keep* the theme names even if they are useless
  to avoid future conflicts.
2019-03-16 20:59:33 +01:00
ThibG 6b0eda14a1 Avoid race condition when streaming deleted statuses (#10280)
* Avoid race condition when streaming deleted statuses

* Move redis lock to DistributionWorker to avoid extra Redis value
2019-03-16 20:18:47 +01:00
Thibaut Girka 4df3b7cb06 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/settings/follower_domains_controller.rb
  Removed upstream. Did the same here. Maybe we should not have?
- config/locales/en.yml
  Upstream removed the “Authorized followers” page and associated
  translations. This is too close in the file to our glitch-soc-specific
  “flavour” string. No actual conflict.
- config/locales/ja.yml
  Same as above.
- config/locales/pl.yml
  Same as above.
- config/navigation.rb
  No real conflict. New route added too close to the glitch-soc-specific
  “flavours” one.
- config/webpack/configuration.js
  Upstream refactored the webpack(er) configuration quite a bit.
  Tried to keep up.
- config/webpack/loaders/babel.js
  Upstream refactored the webpack(er) configuration quite a bit.
  Tried to keep up.
  The contents of this file have been moved to package.json.
- config/webpack/shared.js
  Upstream refactored the webpack(er) configuration quite a bit.
  Tried to keep up.
- config/webpacker.yml
  Upstream refactored the webpack(er) configuration quite a bit.
  Tried to keep up.
- jest.config.js
  The contents of this file have been moved to package.json.
- package.json
  Upstream refactored the webpack(er) configuration quite a bit.
  Tried to keep up.
- yarn.lock
  Upstream refactored the webpack(er) configuration quite a bit.
  Tried to keep up.
2019-03-16 14:14:20 +01:00
Eugen Rochko 3ad3223b46
Fix detailed poll validation errors not being returned in the API (#10261)
No more "Owned poll is invalid"
2019-03-15 13:36:38 +01:00
Eugen Rochko ba84b6d4d7
Add `visibility` param to reblog REST API (#9851)
Use async worker for creating reblog notification to improve performance
2019-03-15 04:36:41 +01:00
Thibaut Girka 07eb31eeac Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/views/admin/settings/edit.html.haml:
  Conflict due to how the theming settings change.
2019-03-14 17:17:20 +01:00
Eugen Rochko 51e154f5e8
Admission-based registrations mode (#10250)
Fix #6856
Fix #6951
2019-03-14 05:28:30 +01:00
Thibaut Girka 02062aab55 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/about_controller.rb
- app/controllers/tags_controller.rb
- app/views/about/show.html.haml
- spec/views/about/show.html.haml_spec.rb
2019-03-13 15:16:02 +01:00
Eugen Rochko 9f5b55ad4f
Fix poll update handler calling method was that was not available (#10246)
* Fix poll update handler calling method was that was not available

Fix regression from #10209

* Refactor VoteService

* Refactor ActivityPub::DistributePollUpdateWorker and optimize it

* Fix typo

* Fix typo
2019-03-12 22:58:59 +01:00
Eugen Rochko a41afef80c
Optimize RemoveStatusService by removing n+1 query and using shared inboxes (#10247) 2019-03-12 22:34:55 +01:00
Thibaut Girka 89bee860cd Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/services/post_status_service.rb
  Small conflict due to handling of instance-local toots.
  A subsequent change is required to ensure instance-local polls are not leaked
  through Update.
2019-03-11 11:23:50 +01:00
ThibG 3a92885a86 Support pushing and receiving updates to poll tallies (#10209)
* Process incoming poll tallies update

* Send Update on poll vote

* Do not send Updates for a poll more often than once every 3 minutes

* Include voters in people to notify of results update

* Schedule closing poll worker on poll creation

* Add new notification type for ending polls

* Add front-end support for ended poll notifications

* Fix UpdatePollSerializer

* Fix Updates not being triggered by local votes

* Fix tests failure

* Fix web push notifications for closing polls

* Minor cleanup

* Notify voters of both remote and local polls when those close

* Fix delivery of poll updates to mentioned accounts and voters
2019-03-11 00:49:31 +01:00
Thibaut Girka dfe7322922 Merge branch 'master' into glitch-soc/merge-upstream 2019-03-10 16:48:44 +01:00
ThibG c11dff5049 Reject existing Follows when suspending a remote account (#10230)
* Reject existing Follows when suspending a remote account

Partial fix to #10229

* Add tests
2019-03-10 16:18:58 +01:00
Thibaut Girka 5528719bc9 Merge branch 'master' into glitch-soc/merge-upstream 2019-03-08 00:57:13 +01:00
ThibG 3aaac4f134 Do not allow adding votes to expired polls (#10214)
* Do not allow adding votes to expired polls

* Only validate expires_at on create
2019-03-08 00:54:50 +01:00
ThibG 09c042aa10 Handle StaleObjectError when retrieving polls (#10208) 2019-03-07 15:52:38 +01:00
Thibaut Girka f513317ba2 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/status.rb
- db/schema.rb

Both conflicts are caused by us having extra database columns.
2019-03-05 19:23:16 +01:00
ThibG df5924a1db Do not error out on unsalvageable errors in FetchRepliesService (#10175)
* Do not error out on unsalvageable errors in FetchRepliesService

Fixes #10152

* Fix FetchRepliesWorker erroring out on deleted statuses
2019-03-05 15:21:14 +01:00
ThibG d785497ba5 Fix suspended account's fields being set as empty dict instead of list (#10178)
Fixes #10177
2019-03-05 15:19:54 +01:00
Eugen Rochko 4037b5eb1e
Fix last_fetched_at not being set on polls (#10170) 2019-03-05 04:10:01 +01:00
Eugen Rochko e6900b167b
Fix another typo in ActivityPub::FetchRemotePollService (#10146) 2019-03-04 00:52:18 +01:00
Eugen Rochko 878a75ba21
Fix typo in ActivityPub::FetchRemotePollService (#10145) 2019-03-04 00:50:56 +01:00
Eugen Rochko ae1b9cf70a
Fix remote poll expiration time (#10144) 2019-03-04 00:44:34 +01:00
ThibG f821eca3b3 Correctly make polls and media mutually exclusive (#10141) 2019-03-04 00:40:21 +01:00
ThibG e13d3792f3 Make sure the poll is created before storing its id (#10142)
* Make sure the poll is created before storing its id

* Fix updating poll results

* Support fetching Question activities from the search bar
2019-03-04 00:39:06 +01:00
Eugen Rochko 230a012f00
Add polls (#10111)
* Add polls

Fix #1629

* Add tests

* Fixes

* Change API for creating polls

* Use name instead of content for votes

* Remove poll validation for remote polls

* Add polls to public pages

* When updating the poll, update options just in case they were changed

* Fix public pages showing both poll and other media
2019-03-03 22:18:23 +01:00
Thibaut Girka 2267eee356 Merge branch 'master' into glitch-soc/merge-upstream 2019-02-28 21:35:53 +01:00
ThibG 9d3c6f1849 Improved remote thread fetching (#10106)
* Fetch up to 5 replies when discovering a new remote status

This is used for resolving threads downwards. The originating
server must add a “replies” attributes with such replies for it to
be useful.

* Add some tests for ActivityPub::FetchRepliesWorker

* Add specs for ActivityPub::FetchRepliesService

* Serialize up to 5 public self-replies for ActivityPub notes

* Add specs for ActivityPub::NoteSerializer

* Move exponential backoff logic to a worker concern

* Fetch first page of paginated collections when fetching thread replies

* Add specs for paginated collections in replies

* Move Note replies serialization to a first CollectionPage

The collection isn't actually paginable yet as it has no id nor
a `next` field. This may come in another PR.

* Use pluck(:uri) instead of map(&:uri) to improve performances

* Fix fetching replies when they are in a CollectionPage
2019-02-28 15:22:21 +01:00
Thibaut Girka 0158937fc2 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- Dockerfile
  Took upstream version and reverted f13ebd02c9.
  Hopefuly it's not needed anymore.
- app/controllers/api/v1/search_controller.rb
  The conflict was due to us raising the number of results returned.
  Upstream raised it further, so took it.
- config/locales/de.yml
  Took upstream changes to theme translation strings.
- config/locales/gl.yml
  Took upstream changes to theme translation strings.
- config/locales/nl.yml
  Took upstream changes to theme translation strings.
- config/locales/sk.yml
  Took upstream changes to theme translation strings.
2019-02-26 17:04:44 +01:00
Eugen Rochko e7f20cc43f
Add type, limit, offset, min_id, max_id, account_id to search API (#10091)
* Add type, limit, offset, min_id, max_id, account_id to search API

Fix #8939

* Make the offset work on accounts and hashtags search as well

* Assure brakeman we are not doing mass assignment here

* Do not allow paginating unless a type is chosen

* Fix search query and index id field on statuses instead of created_at
2019-02-26 15:21:36 +01:00
NOGISAKA Sadata e050958794 Skip some methods in BatchedRemoveStatusService when account is nil (#10095)
* Skip some methods in BatchedRemoveStatusService when account is nil

Sometimes `bin/tootctl accounts cull` fails with below error:

undefined method `followers_for_local_distribution' for nil:NilClass (NoMethodError)

This commit makes BatchedRemoveStatusService to skip below methods when
`account` is nil:

- unpush_from_home_timelines()
- unpush_from_list_timelines()
- batch_stream_entries()

* Fix rubocop error: Use `next` to skip iteration.
2019-02-24 14:28:36 +01:00
Thibaut Girka 06cc04fd23 Merge branch 'master' into glitch-soc/merge-upstream 2019-02-15 18:02:45 +01:00
Eugen Rochko dad339da6d
Filter incoming Create activities by relation to local activity (#10005)
Reject those from accounts with no local followers, from relays
that are not enabled, which do not address local accounts and are
not replies to accounts that do have local followers
2019-02-13 18:42:47 +01:00
Thibaut Girka 0e34e91661 Merge branch 'master' into glitch-soc/merge-upstream 2019-02-12 20:32:48 +01:00
ThibG 99c4b5b75a Move sending account Delete to anyone but the account's followers to the pull̀ queue (#10016) 2019-02-12 05:10:43 +01:00
Thibaut Girka bf94a43496 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/oauth/authorized_applications_controller.rb
  Two changes too close to each other
- app/controllers/settings/sessions_controller.rb
- app/lib/user_settings_decorator.rb
  Two changes too close to each other
- app/models/media_attachment.rb
  New changes too close to glitch-soc only changes.
- app/models/user.rb
  Two changes too close to each other.
- app/services/remove_status_service.rb
  Kept direct timeline code which had been removed upstream.
- app/views/settings/preferences/show.html.haml
  Two changes too close to each other.
- config/locales/en.yml
  Introduction of a new string too close to glitch-soc-only's “flavour”
- config/locales/ja.yml
  Introduction of a new string too close to glitch-soc-only's “flavour”
- config/locales/pl.yml
  Introduction of a new string too close to glitch-soc-only's “flavour”
- config/locales/simple_form.en.yml
  Introduction of a new string too close to glitch-soc-only's “skin”
- config/locales/simple_form.pl.yml
  Introduction of a new string too close to glitch-soc-only's “skin”
- config/settings.yml
  Reverted upstream's decision of enabling posting application by default.
2019-02-10 21:10:09 +01:00
abcang e186bd2fb0 Fix Tombstone.delete_all ArgumentError (#9978) 2019-02-06 02:50:52 +01:00
Eugen Rochko 364f2ff9aa
Add featured hashtags to profiles (#9755)
* Add hashtag filter to profiles

GET /@:username/tagged/:hashtag
GET /api/v1/accounts/:id/statuses?tagged=:hashtag

* Display featured hashtags on public profile

* Use separate model for featured tags

* Update featured hashtag counters on-write

* Limit featured tags to 10
2019-02-04 04:25:59 +01:00
Eugen Rochko d14c276e58
Add option to overwrite imported data (#9962)
* Add option to overwrite imported data

Fix #7465

* Add import for domain blocks
2019-02-03 03:59:51 +01:00
ysksn bcfff65195 Create Redisable#redis (#9633)
* Create Redisable

* Use #redis instead of Redis.current
2019-02-02 19:11:38 +01:00
Thibaut Girka 7251bc3853 Merge branch 'master' into glitch-soc/merge-upstream 2019-01-27 13:57:25 +01:00
Thibaut Girka f938800ef4 Revert to storing application info, and display it to the author via API 2019-01-27 13:56:41 +01:00
Thibaut Girka 05415fa5d3 Make storing and displaying application used to toot opt-in 2019-01-27 13:56:41 +01:00
ThibG e2a5be6e9a Prevent posting toots with media attachments from someone else (#9921) 2019-01-26 23:59:39 +01:00
Thibaut Girka 04e2c67389 Merge branch 'master' into glitch-soc/merge-upstream 2019-01-25 11:37:49 +01:00
Renato "Lond" Cerqueira e1ec3a9f09 Rescue SSL error in verify link service (#9914)
If the first link to be verified contains a rel=me link with a SSL
error, the VerifyAccountLinksWorker will fail and not try the following
links. This rescues the SSL error when fetching the link, avoiding this
issue.
2019-01-24 15:38:18 +01:00
ThibG 061feb63ed Fix scheduled toot with media immediately creating a toot (#9894)
* Add test for not persisting status when attaching media to scheduled toot

* Prevent status used for validation from being persisted to the database

Fixes #9893

Thanks to tateisu for the help investigating this.
2019-01-21 20:03:04 +01:00
Thibaut Girka 9dfbcf941e Merge branch 'master' into glitch-soc/merge-upstream 2019-01-20 16:49:23 +01:00
Eugen Rochko 6ca9015789
Fix error when `scheduled_at` attribute is malformed (#9866) 2019-01-19 22:01:01 +01:00
Thibaut Girka 5e0cf92fd1 Merge branch 'master' into glitch-soc/merge-upstream
No conflicts.
2019-01-19 18:28:37 +01:00
ThibG aeb124491d Reject existing Follow in addition to sending a Block (#9811)
Mastodon expects remote servers to remove follow relationships upon receiving
a Block. However, the spec only evokes Block activities in a C2S context, never
in a S2S context.

This PR, in addition to federating the Block, explicitly sends a Reject for any
affected follow relationship, which makes a bit more sense with regards to the
spec.
2019-01-18 15:57:19 +01:00
ThibG 75b1488cf4 Add tombstones for remote statuses (#9830)
* Add Tombstone model to remember object deletion

* Do not recreate a status if it has been deleted

* Record Tombstone for remote deleted items

Also, only record deleted items from same-host actors

* Clear an user's tombstones when their key change
2019-01-18 15:56:55 +01:00
Eugen Rochko 69f782b54d
Fix code style of regeneration-related code (#9843) 2019-01-18 01:02:51 +01:00
Thibaut Girka dbe311f0e9 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- config/locales/simple_form.pl.yml
2019-01-16 14:20:07 +01:00
Renato "Lond" Cerqueira 5c5e14c816 Fix undefined method error in sidekiq (#9807)
* Fix undefined method error in sidekiq

Body can be not nil but still be empty, which causes a
`NoMethodError: undefined method `[]' for nil:NilClass` further in the
code. This checks for an empty body to avoid the issue.

* Fix codeclimate issue
2019-01-14 17:28:41 +01:00
Thibaut Girka a2a64ecd3e Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- .eslintrc.yml
  Removed, as upstream removed it.
- app/controllers/admin/statuses_controller.rb
  Minor code cleanup when porting one of our features.
- app/models/account.rb
  Note length validation has changed upstream.
  We now use upstream's validation (dropped legacy glitch-soc
  account metadata stuff) but with configurable limit.
- app/services/post_status_service.rb
  Upstream has added support for scheduled toots, refactoring
  the code a bit. Adapted our changes to this refactoring.
- app/views/stream_entries/_detailed_status.html.haml
  Not a real conflict, changes too close.
- app/views/stream_entries/_simple_status.html.haml
  Not a real conflict, changes too close.
2019-01-10 21:00:30 +01:00
Thibaut Girka fb0c906c71 Revert "Revert "Add handler for Move activity (#9629)""
This reverts commit bb96a74637.
2019-01-10 18:46:17 +01:00
ThibG 28b482874a Improvements to signature verification (#9667)
* Refactor signature verification a bit

* Rescue signature verification if recorded public key is invalid

Fixes #8822

* Always re-fetch AP signing key when HTTP Signature verification fails

But when the account is not marked as stale, avoid fetching collections and
media, and avoid webfinger round-trip.

* Apply stoplight to key/account update as well as initial key retrieval
2019-01-07 21:45:13 +01:00
Eugen Rochko 43c61bca60
Add locale param to sign-up API (#9747)
Fix #9627
2019-01-07 14:50:20 +01:00
Eugen Rochko 1cbdf8d218
Fix wrong param name in scheduled statuses and return params in API (#9725)
The database column and API param are called in_reply_to_id, not
in_reply_to_status_id, so it makes no sense to encode it that way
2019-01-06 12:03:27 +01:00
Eugen Rochko a49d43d112
Add scheduled statuses (#9706)
Fix #340
2019-01-05 12:43:28 +01:00
Naoki Kosaka fae32634b1 Use Contact User as Relay, Report, Subscribe. (#9661)
* Use Contact User as Relay, Report, Subscribe.

* Use Account.representative to fetch contact user.

* Use find_local.

* No reason to use Account.representative in subscribe_service.

* Don't required representative!

* Fallback is included in Account.representative method.
2019-01-05 07:17:12 +01:00
Thibaut Girka bb96a74637 Revert "Add handler for Move activity (#9629)"
This reverts commit 0f938ff29c.
2019-01-02 15:36:59 +01:00
Thibaut Girka 571d219bb9 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts manually resolved:
- app/services/post_status_service.rb
- config/locales/simple_form.pl.yml
- config/routes.rb
- config/webpack/loaders/sass.js
- config/webpack/shared.js
- package.json
- yarn.lock
2019-01-02 15:36:53 +01:00
kodai 6a090db4c9 Fix links not being verified (#9673)
Fix #9365
2018-12-31 15:29:10 +01:00
Jeong Arm 511231ce57 Skip full text search if failed (#9654)
* Skip full text search if failed

* Move rescue block to "perform_statuses_search!"
2018-12-30 19:00:29 +01:00
ThibG 6fb6a53938 Avoid duplicate work by merging ReplyDistributionWorker into DistributionWorker (#9660) 2018-12-30 19:00:04 +01:00
ThibG 290932602b Reduce usage of LD signatures (#9659)
* Do not LDS-sign Follow, Accept, Reject, Undo, Block

* Do not use LDS for Create activities of private toots

* Minor cleanup

* Ignore unsigned activities instead of misattributing them

* Use status.distributable? instead of querying visibility directly
2018-12-30 09:48:59 +01:00
Eugen Rochko 0f938ff29c
Add handler for Move activity (#9629) 2018-12-29 02:24:36 +01:00
Eugen Rochko 6e49907ecf
Improve admin UI for account view (#9643) 2018-12-28 03:38:41 +01:00
Eugen Rochko 5d2fc6de32
Add REST API for creating an account (#9572)
* Add REST API for creating an account

The method is available to apps with a token obtained via the client
credentials grant. It creates a user and account records, as well as
an access token for the app that initiated the request. The user is
unconfirmed, and an e-mail is sent as usual.

The method returns the access token, which the app should save for
later. The REST API is not available to users with unconfirmed
accounts, so the app must be smart to wait for the user to click a
link in their e-mail inbox.

The method is rate-limited by IP to 5 requests per 30 minutes.

* Redirect users back to app from confirmation if they were created with an app

* Add tests

* Return 403 on the method if registrations are not open

* Require agreement param to be true in the API when creating an account
2018-12-24 19:12:38 +01:00
ThibG 5f387995d9 Limit maximum visibility of local silenced users to unlisted (#9583)
Fixes #9580
2018-12-24 19:06:14 +01:00
Thibaut Girka 034ffc079e Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/admin/base_controller.rb
  Some refactoring made upstream, no real conflict.
- app/javascript/mastodon/features/compose/components/compose_form.js
  Updated using upstream's code but using maxChars instead of the
  hardcoded length of 500 characters per toot.
- app/javascript/styles/mastodon/components.scss
  Upstream redesigned the onboarding modal. Not sure why we had a
  conflict there.
2018-12-18 16:55:15 +01:00
ThibG e709b8da0d Ignore low-confidence CharlockHolmes guesses when parsing link cards (#9510)
* Add failing test for windows-1251 link cards

* Ignore low-confidence CharlockHolmes guesses

Fixes #9466

* Fix no method error when charlock holmes cannot detect charset
2018-12-17 19:19:45 +01:00
Thibaut Girka 65e994b29b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/directories_controller.rb
- app/controllers/settings/applications_controller.rb
- app/controllers/settings/base_controller.rb
- app/controllers/settings/deletes_controller.rb
- app/controllers/settings/exports_controller.rb
- app/controllers/settings/follower_domains_controller.rb
- app/controllers/settings/imports_controller.rb
- app/controllers/settings/migrations_controller.rb
- app/controllers/settings/notifications_controller.rb
- app/controllers/settings/preferences_controller.rb
- app/controllers/settings/sessions_controller.rb
- app/controllers/settings/two_factor_authentication/confirmations_controller.rb
- app/controllers/settings/two_factor_authentication/recovery_codes_controller.rb
- app/controllers/settings/two_factor_authentications_controller.rb

Conflicts were due to some refactoring already made in glitch-soc
when introducing flavours.
2018-12-15 10:45:53 +01:00
Sumit Khanna 769c2d2680 Error message for avatar image that's too large. #9204 (#9518)
* Error message for avatar image that's too large. #9204

* Code climate/formatting

* Removed avatar error message

* Moved valid image dimentions check to update service

* removed unnescessary begin block

* code climate formatting

* code climate indent fix
2018-12-14 05:07:21 +01:00
ysksn 83975a2394 Remove redundant (#9503) 2018-12-12 05:54:34 +01:00
ysksn 351ed12b2f Use find_or_initialize_by (#9504) 2018-12-12 05:54:09 +01:00
Thibaut Girka e7f1bfdc2d Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/javascript/packs/public.js
- app/models/user.rb
- config/settings.yml
- db/schema.rb

Moved public.js changes to settings.js.
2018-12-09 16:08:04 +01:00
Eugen Rochko 73be8f38c1
Add profile directory (#9427)
Fix #5578
2018-12-06 17:36:11 +01:00
Thibaut Girka 1df392819d Merge branch 'master' into glitch-soc/merge-upstream 2018-12-06 12:44:38 +01:00
Eugen Rochko 6ddf0432e7
Improve account suspension speed and completeness (#9290)
- Some associations were missing from the clean-up
- Some attributes were not reset on suspension
- Skip federation and streaming deletes when purging a dead domain
- Move account association definitions to concern
2018-12-03 01:32:08 +01:00
Thibaut Girka f8e07ca5cd Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/status.rb

Resolved by taking both changes (not a real conflict, just changes too close
to each other).
2018-11-27 13:23:02 +01:00
valerauko 824497fbce Ignore JSON-LD profile in mime type comparison (#9179)
Ignore JSON-LD profile in mime type comparison
2018-11-22 12:49:07 +01:00
James Kiesel 6a65d87f4d Simplify hashtag query service, perf increase for 'any' tags (#9298) 2018-11-19 12:49:50 +01:00
Eugen Rochko 8069fd636b
Remove intermediary arrays when creating hash maps from results (#9291) 2018-11-16 15:02:18 +01:00
Thibaut Girka bfe2b9cc50 Merge branch 'master' into glitch-soc/merge-upstream 2018-11-13 16:28:07 +01:00
ThibG 7f5a4be580 Fix emoji update date processing (#9255) 2018-11-10 23:59:51 +01:00
ThibG 9b89c62d43 Check that twitter:player is valid before using it (#9254)
Fixes #9251
2018-11-10 20:42:04 +01:00
Thibaut Girka 9201398507 Merge branch 'master' into glitch-soc/merge-upstream 2018-11-09 14:56:31 +01:00
Eugen Rochko 6d59dfa15d
Optimize the process of following someone (#9220)
* Eliminate extra accounts select query from FollowService

* Optimistically update follow state in web UI and hide loading bar

Fix #6205

* Asynchronize NotifyService in FollowService

And fix failing test

* Skip Webfinger resolve routine when called from FollowService if possible

If an account is ActivityPub, then webfinger re-resolving is not necessary
when called from FollowService. Improve options of ResolveAccountService
2018-11-08 21:05:42 +01:00
Thibaut Girka 36dab188fd Merge branch 'master' into glitch-soc/merge-upstream 2018-11-06 15:22:44 +01:00
m.b cd41c2c6ad Update resolve_url_service.rb (#9188) 2018-11-05 18:54:07 +01:00
James Kiesel 4c03e05a4e Allow joining several hashtags in a single column (#8904)
* Nascent tag menu on frontend

* Hook up frontend to search

* Tag intersection backend first pass

* Update yarnlock

* WIP

* Fix for tags not searching correctly

* Make radio buttons function

* Simplify radio buttons with modeOption

* Better naming

* Rearrange options

* Add all/any/none functionality on backend

* Small PR cleanup

* Move to service from scope

* Small cleanup, add proper service tests

* Don't use send with user input :D

* Set appropriate column header

* Handle auto updating timeline

* Fix up toggle function

* Use tag value correctly

* A bit more correct to use 'self' rather than 'all' in status scope

* Fix some style issues

* Fix more code style issues

* Style select dropdown more better

* Only use to_id'ed value to ensure no SQL injection

* Revamp frontend to allow for multiple selects

* Update backend / col header to account for more flexible tagging

* Update brakeman ignore

* Codeclimate suggestions

* Fix presenter tag_url

* Implement initial PR feedback

* Handle additional tag streaming

* CodeClimate tweak
2018-11-05 18:53:25 +01:00
Thibaut Girka 75b48dc177 Merge branch 'master' into glitch-soc/merge-upstream 2018-10-31 10:08:36 +01:00
Eugen Rochko ce2ee68b64
Revert "Fix FetchAtomService content type handling (#9132)" (#9171)
This reverts commit c36a4a1617.
2018-10-31 00:43:34 +01:00
Thibaut Girka 1c448d6474 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/auth/sessions_controller.rb

Upstream reverted something we partially reverted already.
Reverted the rest to match upstream.
2018-10-30 17:52:08 +01:00
valerauko c36a4a1617 Fix FetchAtomService content type handling (#9132)
* Add profile to json+ld in Accept

It's required by the ActivityPub spec

* Use headers['Content-type'] instead of mime_type

mime_type strips the profile from the content type, but it's still available raw in the headers hash

* Add test for ld+json with profile
2018-10-30 15:07:57 +01:00
Eugen Rochko be202f9377
Accept the same payload in multiple inboxes and deliver (#9150) 2018-10-30 15:03:55 +01:00
Eugen Rochko 47b8d195e6
Always let through notifications from staff (#9152)
* Always let through notifications from staff

Follow-up to #8993

* Let messages from staff through, but no other notifications
2018-10-30 15:02:55 +01:00
ThibG a03d506626 Fix Pleroma mentions being fetched as preview cards (#9158) 2018-10-30 15:02:24 +01:00
Thibaut Girka 94ad76a5b9 Merge branch 'master' into glitch-soc/merge-upstream 2018-10-30 11:22:25 +01:00
ThibG 33a71e8f7c Do not hide boost notifications from followed people with hidden boosts (#9147)
* Do not hide boost notifications from followed people with hidden boosts

Not displaying boosts from a followed user in the Home timeline and not
having notifications when they reblog your own content are two very
separate concerns, tying them together seem counter-intuitive and unwanted.

* Update specs accordingly
2018-10-30 00:47:31 +01:00
Thibaut Girka b00f60f1d3 Merge branch 'master' into glitch-soc/merge-upstream 2018-10-28 08:37:49 +01:00
Eugen Rochko 795f0107d2
Include preview cards in status entity in REST API (#9120)
* Include preview cards in status entity in REST API

* Display preview card in-stream

* Improve in-stream display of preview cards
2018-10-28 06:35:03 +01:00
Thibaut Girka c13cfeac09 Merge branch 'master' into glitch-soc/merge-upstream 2018-10-27 18:54:26 +02:00
Eugen Rochko eef8d9a5f7
Add locality check to ActivityPub::FetchRemoteAccountService (#9109)
* Add locality check to ActivityPub::FetchRemoteAccountService

Fix #8643

Because there are a few places where it is called, it is difficult
to confirm if they all previously checked it for locality. It's better
to make sure within the service.

* Remove faux-remote duplicates of local accounts
2018-10-26 23:08:34 +02:00
Thibaut Girka 5bb8563f6c Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/admin/base_controller.rb
- app/controllers/filters_controller.rb
- app/controllers/invites_controller.rb
- app/controllers/settings/deletes_controller.rb
- app/controllers/settings/exports_controller.rb
- app/controllers/settings/follower_domains_controller.rb
- app/controllers/settings/migrations_controller.rb
- app/controllers/settings/notifications_controller.rb
- app/controllers/settings/preferences_controller.rb
- app/controllers/settings/two_factor_authentication/recovery_codes_controller.rb
- app/javascript/packs/public.js
- app/views/settings/profiles/show.html.haml

Conflicts were mostly due to the addition of body classes to the settings page,
this was caused by rejecting upstream changes for most of those files and
modifying Settings::BaseController instead.

Another cause of conflicts was the deletion of client-side checking of
display name / bio length, this was modified in app/javascript/core/settings.js
instead.
2018-10-26 20:41:43 +02:00
abcang 1e2695198a Skip link-back check if body is nil (#9107) 2018-10-26 03:31:23 +02:00
ThibG 7fee968e9f Do not fetch preview card for mentioned users (#6934) 2018-10-25 18:13:19 +02:00
Thibaut Girka ed8d06212f Keep back own DMs in Home timeline
Follow-up to e45a6edd65, fixes inconsistencies
in glitch-soc behavior.
2018-10-23 20:03:35 +02:00
Thibaut Girka e45a6edd65 Keep new DMs in home feeds and in the old DM timeline
Revert server-side part of 87fdd139b8
2018-10-22 18:15:51 +02:00
Thibaut Girka dcded13a99 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- .github/ISSUE_TEMPLATE/bug_report.md
  Took our version.
- CONTRIBUTING.md
  Updated the embedded copy of upstream's version.
- README.md
  Took our version.
- app/policies/status_policy.rb
  Not a real conflict, took code from both.
- app/views/layouts/embedded.html.haml
  Added upstream's changes (dns-prefetch) and fixed
  `%body.embed`
- app/views/settings/preferences/show.html.haml
  Reverted some of upstream changes, as we have a
  page dedicated for flavours and skins.
- config/initializers/content_security_policy.rb
  Kept our version of the CSP.
- config/initializers/doorkeeper.rb
  Not a real conflict, took code from both.
2018-10-22 17:51:38 +02:00
Thibaut Girka 4739e0f090 Put a video camera emoji or a picture frame emoji instead of “.”
This uses the same logic as the status icons in the glitch flavor.
2018-10-21 16:09:18 +02:00
Thibaut Girka 8729f5e466 Do not move CWs to toot body when toot body is empty
Fixes #395

Instead of leaving the toot body blank, it replaces it with a single “.” in
order for the fold/unfold CW behavior to not look *too* weird on upstream
Mastodon. Note that this does not fix upstream's CW-dropping behavior, as
that is decided at the time the toot is posted, not received.
2018-10-21 16:09:18 +02:00
Eugen Rochko ddd30f331c
Improve support for aspects/circles (#8950)
* Add silent column to mentions

* Save silent mentions in ActivityPub Create handler and optimize it

Move networking calls out of the database transaction

* Add "limited" visibility level masked as "private" in the API

Unlike DMs, limited statuses are pushed into home feeds. The access
control rules between direct and limited statuses is almost the same,
except for counter and conversation logic

* Ensure silent column is non-null, add spec

* Ensure filters don't check silent mentions for blocks/mutes

As those are "this person is also allowed to see" rather than "this
person is involved", therefore does not warrant filtering

* Clean up code

* Use Status#active_mentions to limit returned mentions

* Fix code style issues

* Use Status#active_mentions in Notification

And remove stream_entry eager-loading from Notification
2018-10-17 17:13:04 +02:00
Quint Guvernator f5e2e96e95 always allow DMs from staff (#8993) 2018-10-16 19:55:05 +02:00
Eugen Rochko 87fdd139b8
Do not push DMs into the home feed (#8940)
* Do not push DMs into the home feed

* Show DMs column after sending a DM, if DMs column is not already shown
2018-10-11 01:31:03 +02:00
Thibaut Girka d17844e6d1 Merge branch 'master' into glitch-soc/merge-upstream 2018-10-08 13:51:33 +02:00
Eugen Rochko 774ac47373
Add conversations API (#8832)
* Add conversations API

* Add web UI for conversations

* Add test for conversations API

* Add tests for ConversationAccount

* Improve web UI

* Rename ConversationAccount to AccountConversation

* Remove conversations on block and mute

* Change last_status_id to be a denormalization of status_ids

* Add optimistic locking
2018-10-07 23:44:58 +02:00
Thibaut Girka 00fcdebed7 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	db/migrate/20170716191202_add_hide_notifications_to_mute.rb
	spec/controllers/application_controller_spec.rb

Took our version, upstream changes were only minor style linting.
2018-10-05 15:23:57 +02:00
Eugen Rochko 7fe137d2f7
Fix link verification for remote accounts (#8868) 2018-10-04 15:47:03 +02:00
Thibaut Girka 343acd96cc Merge branch 'master' into glitch-soc/merge-upstream 2018-09-20 18:36:00 +02:00
Eugen Rochko f92f1ee80a
Support link verification with redirects (#8735)
(e.g. URL shortener)
2018-09-20 00:10:35 +02:00
Thibaut Girka 91bef4759f Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	Vagrantfile
	app/javascript/packs/public.js
	app/views/admin/settings/edit.html.haml
	app/views/settings/preferences/show.html.haml
	app/views/settings/profiles/show.html.haml
	config/locales/es.yml
	config/locales/simple_form.en.yml
	config/webpack/configuration.js
	config/webpack/loaders/babel.js
	package.json
	yarn.lock

Split new additions to app/javascript/packs/public.js to
app/javascript/core/settings.js
2018-09-19 21:46:01 +02:00
Yamagishi Kazutoshi 3da1cc7d5e Fix failed profile verification when rel attribute including values other than me (#8733) 2018-09-19 16:47:31 +02:00
Eugen Rochko 68833a50d4
Fix VerifyAccountLinksWorker not being queued (#8721)
UX-wise, people expect that saving the profile will re-check links even without changing fields content. Bug-wise, `@account` was undefined.

Regression from #8703
2018-09-18 23:57:21 +02:00
Eugen Rochko f4d549d300
Redesign forms, verify link ownership with rel="me" (#8703)
* Verify link ownership with rel="me"

* Add explanation about verification to UI

* Perform link verifications

* Add click-to-copy widget for verification HTML

* Redesign edit profile page

* Redesign forms

* Improve responsive design of settings pages

* Restore landing page sign-up form

* Fix typo

* Support <link> tags, add spec

* Fix links not being verified on first discovery and passive updates
2018-09-18 16:45:58 +02:00
Thibaut Girka cd99255698 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	app/controllers/oauth/authorizations_controller.rb

Just two changes being too close to one another.
Took both.
2018-09-11 16:51:26 +02:00
ThibG bd9e47e9be Handle relative URLs when fetching OEmbed/OpenGraph cards (#8669) 2018-09-10 18:26:28 +02:00
Thibaut Girka 6c233a04a0 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	README.md

Discarded upstream changes to the README, kept our version.
2018-09-03 14:33:06 +02:00
Eugen Rochko c593d6df9c
Add preference for report notification e-mails, skip for duplicates (#8559)
If an unresolved report for the same target account already exists,
no new notification is generated
2018-09-02 00:11:58 +02:00
Thibaut Girka 5d647197ec Add bookmarks to archive takeout 2018-08-30 16:29:47 +02:00
Thibaut Girka 53b7bb9d04 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	app/controllers/api/v1/mutes_controller.rb
	config/locales/simple_form.pl.yml
2018-08-27 18:09:28 +02:00
Eugen Rochko cabdbb7f9c
Add CLI task for rotating keys (#8466)
* If an Update is signed with known key, skip re-following procedure

Because it means the remote actor did *not* lose their database

* Add CLI method for rotating keys

    bin/tootctl accounts rotate [USERNAME]

Generates a new RSA key per account and sends out an Update activity
signed with the old key.

* Key rotation: Space out Update fan-outs every 5 minutes per 1000 accounts

* Skip suspended accounts in key rotation
2018-08-26 20:21:03 +02:00
Eugen Rochko 8adf485c0f
Bump interactions count when a follow happens (#8468) 2018-08-26 20:02:45 +02:00
Thibaut Girka be20eb4b2f Merge branch 'master' into glitch-soc/merge-upstream 2018-08-26 18:21:33 +02:00
Eugen Rochko 55c856c522
Delay e-mail notifications by 2 minutes (#8438)
Fix #8430
2018-08-26 16:53:19 +02:00
Thibaut Girka 246c313d45 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	app/controllers/application_controller.rb

Changed instance theme selection by instance flavour selection.
2018-08-24 15:10:34 +02:00
ThibG 42573b76f1 Do not crash if remote custom emoji does not define updated date (fixes #8376) (#8377) 2018-08-23 00:27:58 +02:00
Thibaut Girka e70fc059a9 Merge branch 'master' into glitch-soc/merge-upstream 2018-08-22 20:56:32 +02:00
Eugen Rochko 802cf6a4c5
Improve federated ID validation (#8372)
* Fix URI not being sufficiently validated with prefetched JSON

* Add additional id validation to OStatus documents, when possible
2018-08-22 20:55:14 +02:00
Eugen Rochko 1b282299df
Add favourites to archive takeout (#8351)
Remove experimental key export
2018-08-22 19:33:10 +02:00
Thibaut Girka 8b4abaa90d Merge branch 'master' into glitch-soc/master
Conflicts:
	config/routes.rb

Added the “endorsements” route from upstream.
2018-08-21 18:24:48 +02:00
Eugen Rochko d98de8ada7
Get rid of all batch order warnings (#8334) 2018-08-21 12:25:50 +02:00
Eugen Rochko 6226aa83d7
Increase reach of Delete->Actor activities (#8305)
Fix #7316
2018-08-20 13:28:05 +02:00
Thibaut Girka 88a0395a58 Merge branch 'master' into glitch-soc/merge-upstream 2018-08-19 09:27:18 +02:00
ThibG abc5548cca Do not process outgoing mentions to suspended accounts (#8272) 2018-08-18 19:42:13 +02:00
Thibaut Girka 280d7b1df8 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	app/models/status.rb
	db/migrate/20180528141303_fix_accounts_unique_index.rb
	db/schema.rb

Resolved by taking upstream changes (no real conflicts, just glitch-soc
specific code too close to actual changes).
2018-08-17 17:43:54 +02:00
ThibG 1ee675d68b Use correct activity id in Accept when receiving duplicate Follow (fixes #8218) (#8244) 2018-08-17 14:08:17 +02:00
ThibG af912fb308 Allow accessing local private/DM messages by URL (#8196)
* Allow accessing local private/DM messages by URL

(Provided the user pasting the URL is authorized to see the toot, obviously)

* Fix SearchServiceSpec tests
2018-08-15 19:33:36 +02:00
Thibaut Girka c1c514ca70 Merge branch 'master' into glitch-soc/merge-upstream
Manually-resolved conflicts:
	.circleci/config.yml
	app/controllers/accounts_controller.rb
	app/controllers/auth/passwords_controller.rb
	app/controllers/statuses_controller.rb
	app/javascript/packs/public.js
	app/models/media_attachment.rb
	app/views/stream_entries/_content_spoiler.html.haml
	app/views/stream_entries/_media.html.haml
	config/locales/en.yml
	config/locales/ja.yml
	config/locales/pl.yml
	lib/mastodon/version.rb

Some content from app/javascript/packs/public.js has been split to
app/javascript/core/settings.js.

Translation strings for glitch-soc's keyword mutes were dropped.

Everything else was mostly “take both”.
2018-08-01 00:18:13 +02:00
Eugen Rochko cc94b1d95a Prevent ActivityPub movedTo recursion (#8092)
Fix #8051
2018-07-28 17:14:55 -04:00
Thibaut Girka 7dbed4dab9 Merge branch 'master' into glitch-soc/merge-upstream 2018-07-17 22:05:25 +02:00
Eugen Rochko a23ac107e4
Fix activity:interactions counter to count all interactions (#8037) 2018-07-16 18:35:43 +02:00
Thibaut Girka f26f1145ac Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	Dockerfile
	app/javascript/packs/common.js
	config/webpack/loaders/sass.js
	config/webpack/shared.js
	db/schema.rb
	package.json
	yarn.lock

A lot of the conflicts come from updating webpack.

Even though upstream deleted app/javascript/packs/common.js, I kept
glitch-soc's version as it unifies JS/CSS packs behavior across flavours.

Ported glitch changes to webpack 4.x
2018-07-16 14:50:42 +02:00
Eugen Rochko e55dce3176
Add federation relay support (#7998)
* Add federation relay support

* Add admin UI for managing relays

* Include actor on relay-related activities

* Fix i18n
2018-07-13 02:16:06 +02:00
Thibaut Girka d392020da6 Merge branch 'master' into glitch-soc/tentative-merge
Conflicts:
	README.md
	app/controllers/statuses_controller.rb
	app/lib/feed_manager.rb
	config/navigation.rb
	spec/lib/feed_manager_spec.rb

Conflicts were resolved by taking both versions for each change.
This means the two filter systems (glitch-soc's keyword mutes and tootsuite's
custom filters) are in place, which will be changed in a follow-up commit.
2018-07-09 07:13:59 +02:00
Eugen Rochko 8461cd4bb5
Send undo of boost to original poster if reblog (#7959)
Fix #7874
2018-07-05 20:57:50 +02:00
Eugen Rochko b0968623fa
Fix check for PotentialFriendshipTracker when replying (#7933) 2018-07-03 05:00:44 +02:00
Eugen Rochko da8fe8079e
Re-add follow recommendations API (#7918)
* Re-add follow recommendations API

    GET /api/v1/suggestions

Removed in 8efa081f21 due to Neo4J
dependency. The algorithm uses triadic closures, takes into account
suspensions, blocks, mutes, domain blocks, excludes locked and moved
accounts, and prefers more recently updated accounts.

* Track interactions with people you don't follow

Replying to, favouriting and reblogging someone you're not following
will make them show up in follow recommendations. The interactions
have different weights:

- Replying is 1
- Favouriting is 10 (decidedly positive interaction, but private)
- Reblogging is 20

Following them, muting or blocking will remove them from the list,
obviously.

* Remove triadic closures, ensure potential friendships are trimmed
2018-07-03 01:47:56 +02:00
Thibaut Girka ab5f450700 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	app/models/user.rb

Resolved by adding :default_language to user settings fields
2018-06-21 20:49:57 +02:00
Eugen Rochko a58ec29631
Allow selecting default posting language instead of auto-detect (#7828)
* Allow selecting default posting language instead of auto-detect

* Enable default language setting in credentials API

* Fix form saving
2018-06-17 18:57:31 +02:00
Thibaut Girka db200226b8 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	app/javascript/mastodon/initial_state.js
	db/schema.rb

Upstream added a new field to initial_state.
Not too sure about what happened with db/schema.rb though…
2018-06-15 20:51:39 +02:00
Eugen Rochko ca85658975
Add autofollow option to invites (#7805)
* Add autofollow option to invites

* Trigger CodeClimate rebuild
2018-06-15 18:00:23 +02:00
Eugen Rochko c3b3594305
Remove placeholder text for media-only toots (#7806)
Ref: cfa9b6e13a

This breaks compatibility with pre-2.3.0 Mastodon and older
software, but at the time of writing the network is >80% above
that version.

Compatibility broken only for toots with no text.
2018-06-15 15:51:51 +02:00
Jenkins 4d8d1fa129 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-06-12 13:17:21 +00:00
Eugen Rochko 10f51c9886
Fix domain hiding logic (#7765)
* Send rejections to followers when user hides domain they're on

* Use account domain blocks for "authorized followers" action

Replace soft-blocking (block & unblock) behaviour with follow rejection

* Split sync and async work of account domain blocking

Do not create domain block when removing followers by domain, that
is probably unexpected from the user's perspective.

* Adjust confirmation message for domain block

* yarn manage:translations
2018-06-09 22:46:54 +02:00
David Yip 8142bd2553
Merge remote-tracking branch 'tootsuite/master' into merge-upstream
Conflicts:
 	app/models/status.rb

The conflict in the Status model was due to
5bf5003384.
It was resolved by accepting tootsuite's changes.
2018-06-07 05:13:49 -05:00
Yamagishi Kazutoshi e3fb528d12 Skip processing when HEAD method returns 501 (#7730) 2018-06-04 13:42:53 +02:00
Akihiko Odaki 5bf5003384 Do not mark remote status sensitive even if spoiler text is present (#7395)
Old statuses and statuses from Pawoo, which runs a modified version of
Mastodon, may not have been marked sensitive even if spoiler text is
present.

Such statuses are still not marked sensitve if they are local or
arrived before version upgrade. Marking recently fetched remote status
sensitive contradicts the behavior.

Considering what people expected when they authored such statuses, this
change removes the sensitivity enforcement.
2018-06-04 04:46:14 +02:00
David Yip 3550470c18
Merge remote-tracking branch 'origin/master' into gs-master
Conflicts:
 	app/javascript/mastodon/locales/en.json
 	app/javascript/mastodon/locales/ja.json
 	app/javascript/mastodon/locales/pl.json

The above conflicts appear to be a text conflict introduced by
glitch-soc's additional level of columns (i.e. moving a bunch of columns
under the Misc option).  They were resolved via accept-ours.
2018-06-02 16:15:36 -05:00
Eugen Rochko 1e938b966e
Exclude unlisted, private and direct toots from affecting trends (#7686) 2018-05-31 16:47:28 +02:00
Jenkins 165b5dc7f5 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-05-31 01:17:25 +00:00
Eugen Rochko a7d726c383
Improve counter caches on Status and Account (#7644)
Do not touch statuses_count on accounts table when mass-destroying
statuses to reduce load when removing accounts, same for
reblogs_count and favourites_count

Do not count statuses with direct visibility in statuses_count

Fix #828
2018-05-30 02:50:23 +02:00
Eugen Rochko 461542784b
Reduce wasted work in RemoveStatusService due to inactive followers (#7672) 2018-05-29 22:55:33 +02:00
Thibaut Girka c087738270 Merge branch 'master' into glitch-soc/merge-upstream 2018-05-29 13:52:26 +02:00
Eugen Rochko 9bd23dc4e5
Track trending tags (#7638)
* Track trending tags

- Half-life of 1 day
- Historical usage in daily buckets (last 7 days stored)
- GET /api/v1/trends

Fix #271

* Add trends to web UI

* Don't render compose form on search route, adjust search results header

* Disqualify tag from trends if it's in disallowed hashtags setting

* Count distinct accounts using tag, ignore silenced accounts
2018-05-27 21:45:30 +02:00
Thibaut Girka a4c9bda771 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	app/javascript/styles/mastodon-light.scss
	config/locales/en.yml
	config/locales/fr.yml
	config/locales/simple_form.pl.yml
	config/themes.yml

Conflicts resolved by deleting config/themes.yml,
marking app/javascript/styles/mastodon-light.scss as added,
and taking all new translation strings, not removing anything from
them.
2018-05-21 16:40:02 +02:00
Eugen Rochko 32d4372381
Use #any? instead of #exists? when checking media attachments (#7570)
If media_attachments are not loaded, SQL query is the same, but
the #exists? method performs SQL query even if preloaded
2018-05-21 16:01:16 +02:00
Yamagishi Kazutoshi 7403e5d306 Add media timeline (#6631) 2018-05-21 12:43:38 +02:00
Thibaut Girka 78ac246b87 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	Gemfile

Solved conflicts by using upstream's line for posix-spawn
2018-05-16 21:52:38 +02:00
Akihiko Odaki 55fd55714a Raise Mastodon::RaceConditionError if Redis lock failed (#7511)
An explicit error allows user agents to know the error and Sidekiq to
retry.
2018-05-16 12:29:45 +02:00
unarist d47091eb97 Fix custom emoji handling in UpdateRemoteProfileService (OStatus) (#7501)
This patch fixes NoMethodError and others in RemoteProfileUpdateWorker.
2018-05-15 16:03:34 +02:00
Wiktor 7fe2993b87 Fix account URI when updating ActivityPub account (#7488)
Updates account `uri` field on each call to `update_account` instead of
only once during `create_account` to mirror the same behavior in OStatus
`ResolveAccountService` class [0].

ActivityPub accounts are identified using `@username` and `@domain` pair
instead of URI since #6842.

This fixes #7479: a bug when the account identified by `@username` and
`@domain` changes its URI.

[0]:
03b69ebc45/app/services/resolve_account_service.rb (L121)
2018-05-14 22:56:45 +02:00
Thibaut Girka ba7ee67498 Merge branch 'master' into glitch-soc/merge 2018-05-14 20:51:50 +02:00
abcang 3793e598d0 Call media.present? because media may be nil (#7474) 2018-05-14 12:51:53 +02:00
ThibG 7467361d70 Fetch boosted statuses on behalf of a follower (fixes #7426) (#7459)
When an ActivityPub Announce is processed and the boosted toot is not known,
fetch it on behalf of one of the booster's followers. This is to allow
fetching self-boosts of previously-unknown private toots.

If fetching on behalf of a user fails, try fetching it anonymously: the
selected follower of a boosting user may be banned by the boosted toot's
author.
2018-05-12 16:48:32 +02:00
Thibaut Girka 45fce0e496 Merge branch 'master' into glitch-soc/merge
Conflicts:
	app/controllers/invites_controller.rb
	app/serializers/initial_state_serializer.rb
	config/locales/ko.yml
2018-05-11 18:12:42 +02:00
Eugen Rochko b4fb766b23
Add REST API for Web Push Notifications subscriptions (#7445)
- POST /api/v1/push/subscription
- PUT /api/v1/push/subscription
- DELETE /api/v1/push/subscription
- New OAuth scope: "push" (required for the above methods)
2018-05-11 11:49:12 +02:00
Thibaut Girka 1c9c0167b7 Merge branch 'master' into glitch-soc/master
Conflicts:
	app/models/account.rb
	app/views/accounts/_header.html.haml
2018-05-10 00:03:28 +02:00
Yamagishi Kazutoshi d2ee48977c Rescue Mastodon::LengthValidationError in FetchLinkCardService (#7424) 2018-05-09 08:39:08 +02:00
Surinna Curtis 01dfd6dbc8 Take the first recognized actor_type. (#7410) 2018-05-08 13:30:04 +02:00
ThibG 993e68a7dd Fix hashtags not being federated on mentions (fixes #6900) (#7406) 2018-05-08 03:36:59 +02:00
Eugen Rochko 42cd363542
Bot nameplates (#7391)
* Store actor type in database

* Add bot nameplate to web UI, add setting to preferences, API, AP
Fix #7365

* Fix code style issues
2018-05-07 09:31:07 +02:00
Eugen Rochko 0f0cc3f2eb
Support explicitly supplying language code for status via REST API (#7389) 2018-05-07 09:30:53 +02:00
Eugen Rochko c947e2e4c5
Fix handling of malformed ActivityPub payloads when URIs are nil (#7370)
* Fix handling of malformed ActivityPub payloads when URIs are nil

* Gracefully handle JSON-LD canonicalization failures
2018-05-05 18:22:34 +02:00
Eugen Rochko c73ce7b695
Store home feeds for 7 days instead of 14 (#7354)
* Store home feeds for 7 days instead of 14

Reduces workload for status fan-out to active followers

* Fix test for user model
2018-05-05 00:54:24 +02:00
David Yip c816701550
Merge remote-tracking branch 'origin/master' into gs-master
Conflicts:
 	.travis.yml
 	Gemfile.lock
 	README.md
 	app/controllers/settings/follower_domains_controller.rb
 	app/controllers/statuses_controller.rb
 	app/javascript/mastodon/locales/ja.json
 	app/lib/feed_manager.rb
 	app/models/media_attachment.rb
 	app/models/mute.rb
 	app/models/status.rb
 	app/services/mute_service.rb
 	app/views/home/index.html.haml
 	app/views/stream_entries/_simple_status.html.haml
 	config/locales/ca.yml
 	config/locales/en.yml
 	config/locales/es.yml
 	config/locales/fr.yml
 	config/locales/nl.yml
 	config/locales/pl.yml
 	config/locales/pt-BR.yml
 	config/themes.yml
2018-05-03 17:23:44 -05:00
Eugen Rochko 658cbc9425
Improve PostStatusService performance (#7317)
Offload creation of local notifications to a worker. Remove two
redundant SQL queries from ProcessMentionsService, remove n+1
XML/JSON serialization via memoization
2018-05-02 22:10:57 +02:00
Eugen Rochko cb5b5cb5f7
Slightly reduce RAM usage (#7301)
* No need to re-require sidekiq plugins, they are required via Gemfile

* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile

* Replace ruby-oembed with FetchOEmbedService

Reduce startup by 45382 allocated objects

* Remove preloaded JSON-LD in favour of caching HTTP responses

Reduce boot RAM by about 6 MiB

* Fix tests

* Fix test suite by stubbing out JSON-LD contexts
2018-05-02 18:58:48 +02:00
abcang 71a7cea73f Keep notification when muting_notifications is true (#7311)
* Keep notification when muting_notifications is true

* Retrun mute object

* Fix test
2018-05-02 16:14:51 +02:00
Eugen Rochko f62539ce5c
Remove most behaviour disparities between blocks and mutes (#7231)
* Remove most behaviour disparities between blocks and mutes

The only differences between block and mute should be:

- Mutes can optionally NOT affect notifications
- Mutes should not be visible to the muted

Fix #7230
Fix #5713

* Do not allow boosting someone you blocked

Fix #7248

* Do not allow favouriting someone you blocked

* Fix nil error in StatusPolicy
2018-05-02 15:50:20 +02:00
Eugen Rochko 965345316f
Guard against nil URLs in Request class (#7284)
Fix #7265
2018-05-02 15:44:22 +02:00
Surinna Curtis dc786c0cf4 Support Actors/Statuses with multiple types (#7305)
* Add equals_or_includes_any? helper in JsonLdHelper

* Support arrays in JSON-LD type fields for actors/tags/objects.

* Spec for resolving accounts with extension types

* Style tweaks for codeclimate
2018-05-02 12:40:24 +02:00
Emelia Smith 495303d9b8 Prevent suspended accounts from appearing in AccountSearchService (#7246) 2018-04-23 21:27:18 +02:00
Eugen Rochko 75c4ab9d12
Remove "nsfw" category for sensitive statuses in OStatus serializer (#7048)
Fix #7011
2018-04-22 22:09:03 +02:00
David Yip 16d5217502
Merge remote-tracking branch 'origin/master' into gs-master
Conflicts:
 	app/controllers/home_controller.rb
 	app/controllers/stream_entries_controller.rb
 	app/javascript/mastodon/locales/ja.json
 	app/javascript/mastodon/locales/pl.json
2018-04-18 18:48:12 -05:00
Eugen Rochko ff87d1bc3e
Rescue SSL errors when processing mentions, remove useless line (#7184) 2018-04-19 00:53:31 +02:00
Kaito Sinclaire 156b916caf Direct messages column (#4514)
* Added a timeline for Direct statuses
* Lists all Direct statuses you've sent and received
* Displayed in Getting Started
* Streaming server support for direct TL

* Changes to match other timelines in 2.0
2018-04-18 13:09:06 +02:00
ThibG 1c379b7ef4 Remove extra spaces from search API queries and public account headers (fixes #7129) (#7152) 2018-04-16 17:19:04 +02:00
David Yip 813da6788e
Merge remote-tracking branch 'origin/master' into gs-master
Conflicts:
 	db/schema.rb
2018-04-14 09:12:50 -05:00
Eugen Rochko 78ed4ab75f
Add bio fields (#6645)
* Add bio fields

- Fix #3211
- Fix #232
- Fix #121

* Display bio fields in web UI

* Fix output of links and missing fields

* Federate bio fields over ActivityPub as PropertyValue

* Improve how the fields are stored, add to Edit profile form

* Add rel=me to links in fields

Fix #121
2018-04-14 12:41:08 +02:00
Jenkins e3d0d72b29 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-04-14 09:17:23 +00:00
Eugen Rochko 778562c223
Ensure SynchronizeFeaturedCollectionWorker is unique and clean up (#7043)
* Ensure SynchronizeFeaturedCollectionWorker is unique and clean up

Fix #7041

* Fix code style issue
2018-04-13 01:27:22 +02:00
Jenkins 33c2bbdda7 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-04-11 00:50:09 +00:00
Renato "Lond" Cerqueira 0c52654b52 When creating status, if no sensitive status is given, use default (#7057)
Clients using the API that do not provide the sensitive flag are always
posting with false sensitive option.
2018-04-09 23:02:42 +02:00
Jenkins 6611100480 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-04-03 19:17:20 +00:00
Nolan Lawson 6ff3b3e4db Fix nil account issue in ProcessAccountService (#7019) 2018-04-03 13:08:11 +02:00
David Yip eb39db7f93
Merge remote-tracking branch 'origin/master' into gs-master
Conflicts:
 	app/javascript/mastodon/locales/en.json
 	app/javascript/mastodon/locales/ja.json
 	app/javascript/mastodon/locales/pl.json
 	app/views/accounts/_header.html.haml
2018-04-01 20:24:50 -05:00
Eugen Rochko f890d2a766
Support all ActivityPub actor types (#6997)
Fix #6973
2018-04-02 02:10:53 +02:00
David Underwood 123a343d11 [WIP] Enable custom emoji on account pages and in the sidebar (#6124)
Federate custom emojis with accounts
2018-04-01 23:55:42 +02:00
Jenkins f0bb3ff533 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-03-30 19:17:17 +00:00
ThibG e573bb0990 Fix compatibility with PeerTube (#6968)
* Support fetching objects of convertible types by URL (fixes #6924)

* Ignore invalid hashtags
2018-03-30 15:44:54 +02:00
David Yip f61aa8e0f7
Merge remote-tracking branch 'origin/master' into gs-master
Conflicts:
 	app/javascript/styles/mastodon/components.scss
 	app/models/media_attachment.rb
2018-03-27 10:26:47 -05:00
Akihiko Odaki 40e5d2303b Validate HTTP response length while receiving (#6891)
to_s method of HTTP::Response keeps blocking while it receives the whole
content, no matter how it is big. This means it may waste time to receive
unacceptably large files. It may also consume memory and disk in the
process. This solves the inefficency by checking response length while
receiving.
2018-03-26 14:02:10 +02:00
Jenkins 837b3804bf Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-03-25 15:17:21 +00:00
Akihiko Odaki 54b273bf99 Close http connection in perform method of Request class (#6889)
HTTP connections must be explicitly closed in many cases, and letting
perform method close connections makes its callers less redundant and
prevent them from forgetting to close connections.
2018-03-24 12:49:54 +01:00
Jenkins e8aeec4fea Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-03-20 19:17:20 +00:00
Eugen Rochko 9381a7d9d5
Use username/domain to match existing accounts in ActivityPub (#6842)
See also: #6837, #6667
2018-03-20 14:57:46 +01:00
Jenkins 9341b982b6 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-03-17 17:17:21 +00:00
nightpool 566ace2d64 Add entropy to download filenames (#6811)
pretty quick fix, and with the 1 week expiration i don't think we need to be too worried about the existing files

closes #6798
2018-03-17 17:39:28 +01:00
Jenkins 447d7e6127 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-03-09 00:17:17 +00:00
Eugen Rochko 510c9049c7
For now, put a "." into no-text statuses with media for backcompat (#6691) 2018-03-08 08:20:49 +01:00
Eugen Rochko 64db9ed5f6
After blocking domain with reject_media, invalidate cache (#6679)
Media attachments are part of the association cache of statuses,
since they are presumed to be immutable. Unless this cache is
cleared manually, the statuses will continue to look like they
have media embedded.
2018-03-08 06:59:42 +01:00
Eugen Rochko cfa9b6e13a
Remove text requirement when media attached from statuses (#6672) 2018-03-07 08:28:52 +01:00
Eugen Rochko 9110db41c5
Federate pinned statuses over ActivityPub (#6610)
* Federate pinned statuses over ActivityPub

* Display pinned toots in web UI

Fix #6117

* Fix migration

* Fix tests

* Update outbox_serializer.rb

* Update remove_serializer.rb

* Update add_serializer.rb

* Update fetch_featured_collection_service.rb
2018-03-04 09:19:11 +01:00
Aboobacker MK 49092945ab Fix 500 while searching after deleting a post (#6604)
Fixes #6602
2018-03-03 19:45:06 +01:00
David Yip 1b8fcd4df5
Merge remote-tracking branch 'origin/master' into merge-upstream
Conflicts:
 	README.md
 	app/controllers/follower_accounts_controller.rb
 	app/controllers/following_accounts_controller.rb
 	app/serializers/rest/instance_serializer.rb
 	app/views/stream_entries/_simple_status.html.haml
 	config/locales/simple_form.ja.yml
2018-03-02 21:46:44 -06:00
Eugen Rochko 41a01bec23
Federated reports (#6570)
* Fix #2176: Federated reports

* UI for federated reports

* Add spec for ActivityPub Flag handler

* Add spec for ReportService
2018-02-28 06:54:55 +01:00
imncls bb6988a7ac
Merge branch 'master' of https://github.com/tootsuite/mastodon
# Conflicts:
#	app/controllers/settings/exports_controller.rb
#	app/models/media_attachment.rb
#	app/models/status.rb
#	app/views/about/show.html.haml
#	docker_entrypoint.sh
#	spec/views/about/show.html.haml_spec.rb
2018-02-23 23:28:31 +09:00
Eugen Rochko 61ed133fea
Account archive download (#6460)
* Fix #201: Account archive download

* Export actor and private key in the archive

* Optimize BackupService

- Add conversation to cached associations of status, because
  somehow it was forgotten and is source of N+1 queries
- Explicitly call GC between batches of records being fetched
  (Model class allocations are the worst offender)
- Stream media files into the tar in 1MB chunks
  (Do not allocate media file (up to 8MB) as string into memory)
- Use #bytesize instead of #size to calculate file size for JSON
  (Fix FileOverflow error)
- Segment media into subfolders by status ID because apparently
  GIF-to-MP4 media are all named "media.mp4" for some reason

* Keep uniquely generated filename in Paperclip::GifTranscoder

* Ensure dumped files do not overwrite each other by maintaing directory partitions

* Give tar archives a good name

* Add scheduler to remove week-old backups

* Fix code style issue
2018-02-21 23:21:32 +01:00
David Yip 3d033a4687
Merge remote-tracking branch 'tootsuite/master' into merge-upstream
Conflicts:
	Gemfile
	config/locales/simple_form.pl.yml
2018-02-17 00:02:37 -06:00
Eugen Rochko f7765acf9d
Fix #5173: Click card to embed external content (#6471) 2018-02-15 07:04:28 +01:00
Jenkins efc9ffcb80 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-02-11 04:10:16 +00:00
Eugen Rochko 3ebc0ad4d3
Full-text search for authorized statuses (#6423)
* Add full-text search for authorized statuses

- Search API will return statuses that match the query
- Only for logged in users
- Only if you are author of the status,
- Or you were mentioned in it
- Or you favourited or reblogged it
- Configuration over `ES_ENABLED`, `ES_HOST`, `ES_PORT`, `ES_PREFIX`
- Run `rails chewy:deploy` to create & populate index

Fix #5880
Fix #4293
Fix #1152

* Add commented out docker-compose configuration for ES container

* Optimize index import, filter search results

* Add basic normalization to the index

* Add better stemming and normalization to the index

* Skip webfinger request if search query includes both @ and a space

* Fix code style

* Visually separate search result sections

* Fix code style issues
2018-02-09 23:04:47 +01:00
David Yip 4c1fd9a19c
Merge remote-tracking branch 'tootsuite/master' into merge-upstream
Conflicts:
      app/javascript/styles/mastodon/components.scss
2018-02-02 08:39:52 -06:00
abcang 7e5c433dfc Fix saving of oEmbed image (#6409) 2018-02-02 11:57:59 +01:00
Jenkins 0e10667fbe Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-01-24 19:17:11 +00:00
Akihiko Odaki 613e7c7521 Rename ResolveRemoteAccountService to ResolveAccountService (#6327)
The service used to be named ResolveRemoteAccountService resolves local
accounts as well.
2018-01-22 14:25:09 +01:00
Akihiko Odaki 17cecd75ca Rename FetchRemoteResourceService to ResolveURLService (#6328)
The service used to be named FetchRemoteResourceService resolves local
URL as well.
2018-01-22 14:24:22 +01:00
David Yip 1964a0f941
Merge remote-tracking branch 'origin/master' into merge-upstream 2018-01-18 09:17:58 -06:00
Renato "Lond" Cerqueira e56404be41 When must_be_following_dm is on, only notify if recipient dm'ed user (#6283)
* When must_be_following_dm is on, only notify if recipient dm'ed user
Currently, when must_be_following_dm is on, if a user sends a direct
message replying to any status from the recipient, the recipient gets a
notification. This should not be the case, as if the recipient posted
something publicly this can be used to spam their notifications.

* Refactor replied_to_status_is_direct_message?
Following suggestion in PR
2018-01-18 16:12:10 +01:00
David Yip 201e82686f
Merge remote-tracking branch 'origin/master' into merge-upstream
Conflicts:
      app/javascript/styles/mastodon/components.scss
2018-01-17 18:37:09 -06:00
Eugen Rochko 7badad7797
Fix home regeneration (#6251)
* Fix regeneration marker not being removed after completion

* Return HTTP 206 from /api/v1/timelines/home if regeneration in progress
Prioritize RegenerationWorker by putting it into default queue

* Display loading indicator and poll home timeline while it regenerates

* Add graphic to regeneration message

* Make "not found" indicator consistent with home regeneration
2018-01-17 23:56:03 +01:00
Jenkins 0c7dc6c781 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-01-15 06:17:15 +00:00
puckipedia 74c1c9ec01 Allow attributedTo in a status to be an embedded object (#6238) 2018-01-15 06:51:46 +01:00
Jenkins 722b3f567f Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-01-08 04:17:11 +00:00
Eugen Rochko e4a241abef
Fix bad URL schemes being accepted (#6219)
* Fix actors accepting invalid URI schemes or different host between URI and URL

* Fix statuses accepting invalid URI scheme or different host to actor

* Adjust tests to new requirements

* Improve readability of mismatching_origin?/invalid_origin? methods
2018-01-08 05:00:23 +01:00
Jenkins f6adb409fd Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-01-05 22:17:12 +00:00
ThibG a594139115 When fetching an ActivityPub-enabled status, do not re-request it as text/html (#6196) 2018-01-05 22:42:50 +01:00
Jenkins c69a23ae46 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-01-04 23:17:11 +00:00
Eugen Rochko 02ba03d6db
Send one Delete of Actor in ActivityPub when account is suspended (#6172) 2018-01-04 14:40:49 +01:00
Eugen Rochko 7d4f4f9aab
Fix FetchAtomService not finding alternatives if there's a Link header (#6170)
without them, such as is the case with GNU social

Fixes the ability to find GNU social accounts via URL in search and
when using remote follow function
2018-01-04 04:56:04 +01:00
Jenkins 933840bebf Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-01-03 04:17:11 +00:00
Eugen Rochko d319b3dbe4
Update moved-to property when it's removed too (#6160)
* Fix #6140 - Update moved-to property when it's removed too

* Remove trailing whitespace
2018-01-03 00:38:20 +01:00
Jenkins 6d00ca1c71 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-12-22 04:17:14 +00:00
ThibG cea98e0c12 Reduce the number of synchronous resolves when posting toots (#6075) 2017-12-22 02:15:08 +01:00
David Yip 82b2e224a2
Merge branch 'gs-master' into prevent-local-only-federation
Conflicts:
	db/schema.rb
2017-12-15 12:20:56 -06:00
Jenkins 82236a3703 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-12-13 18:17:16 +00:00
Eugen Rochko a8deb6648b
Fix redundant HTTP request in FetchLinkCardService (#6002) 2017-12-13 12:15:28 +01:00
Eugen Rochko 5706fe18c2
Fix #5952 - NameError (regression from #5762) (#5999)
* Fix #5952 - NameError (regression from #5762)

* Fix
2017-12-13 04:12:38 +01:00
Akihiko Odaki f9f6918148 Store preview image for embedded photo in preview cards (#5986)
The preview image would be useful to embed in timeline.
2017-12-12 15:54:38 +01:00
Renato "Lond" Cerqueira 19257d91bf Return false if object does not respond to url (#5988)
Avoid error when the service returns a mostly valid oembed, but has no
url in it, causing a MethodError: undefined method `url'
for #<OEmbed::Response::Photo:0x000056505def9620>
2017-12-12 15:12:09 +01:00
Renato "Lond" Cerqueira fe180f18ff Change conditional to avoid nil into string error in sidekiq (#5987)
* Change conditional to avoid nil into string error in sidekiq
When obtaining information about users with mastodon in a different
subdomain, sidekiq was giving out a 'no implicit conversion of nil into String'

* Use presence instead of blank? with ternary.
Following suggestion on PR
2017-12-12 15:11:13 +01:00
David Yip a057ed5cfe
Merge remote-tracking branch 'tootsuite/master' into merge-upstream 2017-12-12 02:54:13 -06:00
Erin 2efffe77dc reblog_service.rb: Status#local_only -> local_only? 2017-12-11 17:39:11 -06:00
Erin 3ec47e732b post_status_service: stylistic change (local_only -> local_only?) 2017-12-11 15:27:31 -06:00
Erin 288f1293ef set local_only in a before_create callback instead of status service 2017-12-10 21:39:27 -06:00
Erin c1410af368 post_status_service.rb: save the status after setting local_only 2017-12-10 20:35:57 -06:00
Erin 5ef65aab8f replace reblog service check for an 👁️ with #local_only 2017-12-10 17:12:21 -06:00
Erin cfbb95605b set local_only flag on statuses in post_status_service 2017-12-10 17:04:32 -06:00
Andrea Scarpino 6a82939adb Fix account and tag searches with leading/trailing spaces (#5965)
* Strip leading & trailing spaces from account query

* Strip leading & trailing spaces from tag search
2017-12-10 19:35:46 +01:00
Yamagishi Kazutoshi e7d55df38d Ignore HEAD method if does not support (#5949) 2017-12-09 16:53:40 +01:00
Eugen Rochko 86b4d5439c
Fix #5926 - Do not downgrade to OStatus once ActivityPub is known (#5929) 2017-12-09 01:24:47 +01:00
Jenkins 744447b3c0 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-12-07 04:17:14 +00:00
Akihiko Odaki c083816c24 Add embed_url to preview cards (#5775) 2017-12-07 03:37:43 +01:00
Jenkins 8ca91cef45 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-12-06 20:17:13 +00:00
Yamagishi Kazutoshi d7a17b5e8b Search only from followees (#5897) 2017-12-06 11:44:23 +01:00
Yamagishi Kazutoshi b21db9bbde Using double splat operator (#5859) 2017-12-06 11:41:57 +01:00
Eugen Rochko e20895f251
Add list of lists component to web UI (#5811)
* Add list of lists component to web UI

* Add list adding

* Add list removing

* List editor modal

* Add API account search limited by following=true relation

* Rework list editor modal

* Remove mandatory pagination of GET /api/v1/lists/:id/accounts

* Adjust search input placeholder

* Fix rspec (#5890)

* i18n: (zh-CN) Add missing translations for #5811 (#5891)

* i18n: (zh-CN) yarn manage:translations -- zh-CN

* i18n: (zh-CN) Add missing translations for #5811

* Fix some issues

- Display loading/missing state for list timelines
- Order lists alphabetically in overview
- Fix async list editor reset
- Redirect to /lists after deleting unpinned list
- Redirect to / after pinning a list

* Remove dead list columns when a list is deleted or fetch returns 404
2017-12-05 23:02:27 +01:00
Jenkins ad46bc9772 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-30 03:17:12 +00:00
Eugen Rochko 4c6b5dbe96
Add semi-support for Video/Image objects in ActivityPub (#5848)
* Add semi-support for Video/Image objects in ActivityPub

Video and Image objects will create corresponding status records
with manually crafted text contents (title + URL)

* Extract html-url-finding logic into JsonLdHelper

* Fallback to id when url missing, extract supported object types
2017-11-30 04:06:20 +01:00
Eugen Rochko 85e97ecab6
Fix too many forwards (#5854)
* Avoid sending explicit Undo->Announce when original deleted

* Do not forward a reply back to the server that sent it

* Deduplicate inboxes of rebloggers' followers for delete forwarding

* Adjust test

* Fix wrong class, bad SQL, wrong variable, outdated comment
2017-11-30 03:50:05 +01:00
David Yip 95c270f5b1
Merge remote-tracking branch 'origin/master' into gs-master 2017-11-28 11:45:13 -06:00
aschmitz eeaec39888 Allow hiding of reblogs from followed users (#5762)
* Allow hiding of reblogs from followed users

This adds a new entry to the account menu to allow users to hide
future reblogs from a user (and then if they've done that, to show
future reblogs instead).

This does not remove or add historical reblogs from/to the user's
timeline; it only affects new statuses.

The API for this operates by sending a "reblogs" key to the follow
endpoint. If this is sent when starting a new follow, it will be
respected from the beginning of the follow relationship (even if
the follow request must be approved by the followee). If this is
sent when a follow relationship already exists, it will simply
update the existing follow relationship. As with the notification
muting, this will now return an object ({reblogs: [true|false]}) or
false for each follow relationship when requesting relationship
information for an account. This should cause few issues due to an
object being truthy in many languages, but some modifications may
need to be made in pickier languages.

Database changes: adds a show_reblogs column (default true,
non-nullable) to the follows and follow_requests tables. Because
these are non-nullable, we use the existing MigrationHelpers to
perform this change without locking those tables, although the
tables are likely to be small anyway.

Tests included.

See also <https://github.com/glitch-soc/mastodon/pull/212>.

* Rubocop fixes

* Code review changes

* Test fixes

This patchset closes #648 and resolves #3271.

* Rubocop fix

* Revert reblogs defaulting in argument, fix tests

It turns out we needed this for the same reason we needed it in muting:
if nil gets passed in somehow (most usually by an API client not passing
any value), we need to detect and handle it.

We could specify a default in the parameter and then also catch nil, but
there's no great reason to duplicate the default value.
2017-11-28 15:00:35 +01:00
ThibG b8efb5daed Fix handling of temporary failures in ProcessMentionsService (#5842)
* Add test for temporary account resolving failures in ProcessMentionsService

* Fix processing of mentions to already-known remote accounts on temporary failures
2017-11-28 15:00:22 +01:00
Jenkins 8f3e5f6128 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-21 05:17:12 +00:00
abcang 3dce6cbbd7 Fixed duplicating URL of photo type of oEmbed (#5763) 2017-11-20 20:45:54 +01:00
Jenkins a36a2c1796 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-19 15:17:12 +00:00
abcang 2bcc81700c Fix NoMethodError at ActivityPub::FetchRemoteStatusService (#5753) 2017-11-19 15:33:15 +01:00
David Yip d083f7741a Merge remote-tracking branch 'tootsuite/master' 2017-11-19 01:30:29 -06:00
Eugen Rochko 58cede4808
Profile redirect notes (#5746)
* Serialize moved accounts into REST and ActivityPub APIs

* Parse federated moved accounts from ActivityPub

* Add note about moved accounts to public profiles

* Add moved account message to web UI

* Fix code style issues
2017-11-18 19:39:02 +01:00
David Yip 1ab12ba38e Merge remote-tracking branch 'origin/master' into merge-upstream 2017-11-17 17:40:00 -06:00
Eugen Rochko 24cafd73a2
Lists (#5703)
* Add structure for lists

* Add list timeline streaming API

* Add list APIs, bind list-account relation to follow relation

* Add API for adding/removing accounts from lists

* Add pagination to lists API

* Add pagination to list accounts API

* Adjust scopes for new APIs

- Creating and modifying lists merely requires "write" scope
- Fetching information about lists merely requires "read" scope

* Add test for wrong user context on list timeline

* Clean up tests
2017-11-18 00:16:48 +01:00
David Yip b28b405b97 Merge remote-tracking branch 'origin/master' into gs-master 2017-11-16 22:52:37 -06:00
Eugen Rochko 3e90987c8b Fix some rubocop style issues (#5730) 2017-11-17 10:06:26 +09:00
Surinna Curtis 35fbdc36f9 Merge tootsuite/master at 3023725936 2017-11-16 01:21:16 -06:00
Surinna Curtis 031a5a8f92 Optional notification muting (#5087)
* Add a hide_notifications column to mutes

* Add muting_notifications? and a notifications argument to mute!

* block notifications in notify_service from hard muted accounts

* Add specs for how mute! interacts with muting_notifications?

* specs testing that hide_notifications in mutes actually hides notifications

* Add support for muting notifications in MuteService

* API support for muting notifications (and specs)

* Less gross passing of notifications flag

* Break out a separate mute modal with a hide-notifications checkbox.

* Convert profile header mute to use mute modal

* Satisfy eslint.

* specs for MuteService notifications params

* add trailing newlines to files for Pork :)

* Put the label for the hide notifications checkbox in a label element.

* Add a /api/v1/mutes/details route that just returns the array of mutes.

* Define a serializer for /api/v1/mutes/details

* Add more specs for the /api/v1/mutes/details endpoint

* Expose whether a mute hides notifications in the api/v1/relationships endpoint

* Show whether muted users' notifications are muted in account lists

* Allow modifying the hide_notifications of a mute with the /api/v1/accounts/:id/mute endpoint

* make the hide/unhide notifications buttons work

* satisfy eslint

* In probably dead code, replace a dispatch of muteAccount that was skipping the modal with launching the mute modal.

* fix a missing import

* add an explanatory comment to AccountInteractions

* Refactor handling of default params for muting to make code cleaner

* minor code style fixes oops

* Fixed a typo that was breaking the account mute API endpoint

* Apply white-space: nowrap to account relationships icons

* Fix code style issues

* Remove superfluous blank line

* Rename /api/v1/mutes/details -> /api/v2/mutes

* Don't serialize "account" in MuteSerializer

Doing so is somewhat unnecessary since it's always the current user's account.

* Fix wrong variable name in api/v2/mutes

* Use Toggle in place of checkbox in the mute modal.

* Make the Toggle in the mute modal look better

* Code style changes in specs and removed an extra space

* Code review suggestions from akihikodaki

Also fixed a syntax error in tests for AccountInteractions.

* Make AddHideNotificationsToMute Concurrent

It's not clear how much this will benefit instances in practice, as the
number of mutes tends to be pretty small, but this should prevent any
blocking migrations nonetheless.

* Fix up migration things

* Remove /api/v2/mutes
2017-11-15 03:56:41 +01:00
ThibG 1c25853842 Use already-known remote user data if resolving temporarily fails in mentions (#5702) 2017-11-15 01:06:49 +01:00
Eugen Rochko fbef909c2a
Add option to block direct messages from people you don't follow (#5669)
* Add option to block direct messages from people you don't follow

Fix #5326

* If the DM responds to a toot by recipient, allow it through

* i18n: Update Polish translation (for #5669) (#5673)
2017-11-14 21:12:57 +01:00
aschmitz 5128c4261e Updates per code review
Thanks, @valerauko!
2017-11-11 14:37:23 -06:00
aschmitz b95c48748c Per-user reblog hiding implementation/fixes/tests
Note that this will only hide/show *future* reblogs by a user, and does
nothing to remove/add reblogs that are already in the timeline. I don't
think that's a particularly confusing behavior, and it's a lot easier
to implement (similar to mutes, I believe).
2017-11-10 22:04:54 -06:00
aschmitz 4944515020 "Show reblogs" per-follower UI/database changes
TODO:

* Tests (particularly for FollowRequests).
* Anything to respect the setting when putting reblogs in timelines.
2017-11-09 08:41:10 -06:00
ThibG 889ada5ee2 Fix process mentions for local users, as local users are considered to use OStatus (#5618) 2017-11-07 22:15:15 +01:00
ThibG 5d5c0f4f43 Twidere mention workaround (#5552)
* Work around Twidere and Tootdon bug

Tootdon and Twidere construct @user@domain handles from mentions in toots based
solely on the mention text and account URI's domain without performing any
webfinger call or retrieving account info from the Mastodon server.

As a result, when a remote user has WEB_DOMAIN ≠ LOCAL_DOMAIN, Twidere and
Tootdon will construct the mention as @user@WEB_DOMAIN. Now, this will usually
resolve to the correct account (since the recommended configuration is to have
WEB_DOMAIN perform webfinger redirections to LOCAL_DOMAIN) when processing
mentions, but won't do so when displaying them (as it does not go through the
whole account resolution at that time).

This change rewrites mentions to the resolved account, so that displaying the
mentions will work.

* Use lookbehind instead of non-capturing group in MENTION_RE

Indeed, substitutions with the previous regexp would erroneously eat any
preceding whitespace, which would lead to concatenated mentions in the
previous commit.

Note that users will “lose” up to one character space per mention for their
toots, as that regexp is also used to remove the domain-part of mentioned
users for character counting purposes, and it also erroneously removed the
preceding character if it was a space.
2017-11-07 19:08:14 +01:00
Eugen Rochko 1032f3994f
Add ability to disable login and mark accounts as memorial (#5615)
Fix #5597
2017-11-07 19:06:44 +01:00
ThibG 84cfee2488 Do not process undeliverable mentions (#5598)
* Resolve remote accounts when mentioned even if they are already known

This commit reduces the risk of not having up-to-date public key or protocol
information for a remote account, which is required to deliver toots
(especially direct messages).

* Do not add mentions in private messages for remote users we cannot deliver to

Mastodon does not deliver private and direct toots to OStatus users, as there
is no guarantee the remote software understands the toot's privacy. However,
users currently do not get any feedback on it (Mastodon won't attempt delivery,
but the toot will be displayed exactly the same way to the user).

This change introduces *some* feedback by not processing mentions that are
not going to be delivered. A long-term solution is still needed to have
delivery receipts or at least some better indication of what is going on, but
at least an user can see *something* is up.
2017-11-07 14:47:39 +01:00
ThibG 7bea1530f4 Resolve remote accounts when mentioned even if they are already known (#5539)
This commit reduces the risk of not having up-to-date public key or protocol
information for a remote account, which is required to deliver toots
(especially direct messages).
2017-11-07 14:31:57 +01:00
ThibG d37a56c07c Update remote ActivityPub users when fetching their toots (#5545) 2017-10-30 00:24:16 +09:00
ThibG 89c77fe225 Instantiate service classes for each call (fixes #5540) (#5543) 2017-10-27 19:08:30 +02:00
David Yip 42f2045c21 Merge remote-tracking branch 'STJrInuyasha/feature/direct-timeline' into gs-direct-timeline 2017-10-25 16:01:20 -05:00
Matthew Walsh 3db80f75a6 Added a timeline for Direct statuses
* Lists all Direct statuses you've sent and received
* Displayed in Getting Started
* Streaming server support for direct TL
2017-10-22 18:35:14 -07:00
David Yip 42e8c8eb0e Merge tag 'v2.0.0' into gs-master 2017-10-18 11:52:04 -05:00
unarist 3bc8924940 Don't capture scheme-less URLs in the status (#5435)
Specifically, this fixes status length calculation to be same as JS side.

BTW, since this pattern used in not only preview card fetching, we
should extract it (with twitter-regex?) and write tests I think.
2017-10-17 18:32:25 +02:00
David Yip 6cd5b3bbe5 Merge remote-tracking branch 'origin/master' into gs-master 2017-10-16 01:29:02 -05:00
abcang 2eab41cd1a Close connection when succeeded posting (#5390)
* Close connection when succeeded posting

* Update webmock
2017-10-14 14:38:57 +02:00
ThibG 95fe20b78a Fix remote status fetching for “pure” ActivityPub WEB_ACCOUNT users (#5372)
Remote ActivityPub users that have never been known as OStatus users
(whether or not they support it) will not have a “remote_url” attribute
set. In case they reside on an instance with WEB_DOMAIN ≠ LOCAL_DOMAIN,
the current check did rely on “remote_url” to verify the user's domain.
2017-10-13 12:58:13 +02:00
unarist 35ec1c91e3 Fix some failure cases on FetchLinkCardService (#5347)
* If OEmbed response doesn't have a required property `type`, ignore it.
  e.g. `NoMethodError: undefined method 'type' for ...`
* If we failed to detect encoding, fallback to default behavior of Nokogiri.
  e.g. `KeyError: key not found: :encoding`
2017-10-12 12:01:32 +02:00
David Yip edd1a00faf Restore variable assignment in MuteService#call.
291feba6f1 made MuteService return the
result of Account#mute!; this commit restores that behavior.
2017-10-12 04:27:45 -05:00
kibigo! 8d6b9ba494 Merge upstream 2.0ish #165 2017-10-11 10:43:10 -07:00
Eugen Rochko 3888a12c79 Fix #5272 - Order of checks in ActivityPub handler (#5276) 2017-10-08 22:03:34 +02:00
Eugen Rochko 0717d9b3e6 Set snowflake IDs for backdated statuses (#5260)
- Rename Mastodon::TimestampIds into Mastodon::Snowflake for clarity
- Skip for statuses coming from inbox, aka delivered in real-time
- Skip for statuses that claim to be from the future
2017-10-08 17:34:34 +02:00
unarist 5c8ca024ef Improve error handling on LinkCrawlWorker (#5250)
* Improve error handling on LinkCrawlWorker

* Ignore TimeoutError and InvalidURIError too
* Record errors to debug log
* Enable dead job queue on LinkCrawlWorker

Since most of acceptable errors were already ignored, only our side issue should go to dead job queue.

* Ignore all http gem errors
2017-10-06 20:39:08 +02:00
ThibG 2559d9166c Fix regression in FetchRemoteResourceService (#5217)
* Fix regression in FetchRemoteResourceService

* Update specs to match interface changes made in #5114
2017-10-05 00:21:44 +02:00
ThibG 0e1b0f2747 Check Webfinger-returned author URI even when not redirected (#5213)
The whole point of verified_webfinger? is to check the WebFinger-discoverable
URI maps back to the known author URI. This was not actually verified if the
first Webfinger request was not a redirection.
2017-10-04 09:59:28 +02:00
aschmitz 468523f4ad Non-Serial ("Snowflake") IDs (#4801)
* Use non-serial IDs

This change makes a number of nontrivial tweaks to the data model in
Mastodon:

* All IDs are now 8 byte integers (rather than mixed 4- and 8-byte)
* IDs are now assigned as:
  * Top 6 bytes: millisecond-resolution time from epoch
  * Bottom 2 bytes: serial (within the millisecond) sequence number
  * See /lib/tasks/db.rake's `define_timestamp_id` for details, but
    note that the purpose of these changes is to make it difficult to
    determine the number of objects in a table from the ID of any
    object.
* The Redis sorted set used for the feed will have values used to look
  up toots, rather than scores. This is almost always the same as the
  existing behavior, except in the case of boosted toots. This change
  was made because Redis stores scores as double-precision floats,
  which cannot store the new ID format exactly. Note that this doesn't
  cause problems with sorting/pagination, because ZREVRANGEBYSCORE
  sorts lexicographically when scores are tied. (This will still cause
  sorting issues when the ID gains a new significant digit, but that's
  extraordinarily uncommon.)

Note a couple of tradeoffs have been made in this commit:

* lib/tasks/db.rake is used to enforce many/most column constraints,
  because this commit seems likely to take a while to bring upstream.
  Enforcing a post-migrate hook is an easier way to maintain the code
  in the interim.
* Boosted toots will appear in the timeline as many times as they have
  been boosted. This is a tradeoff due to the way the feed is saved in
  Redis at the moment, but will be handled by a future commit.

This would effectively close Mastodon's #1059, as it is a
snowflake-like system of generating IDs. However, given how involved
the changes were simply within Mastodon, it may have unexpected
interactions with some clients, if they store IDs as doubles
(or as 4-byte integers). This was a problem that Twitter ran into with
their "snowflake" transition, particularly in JavaScript clients that
treated IDs as JS integers, rather than strings. It therefore would be
useful to test these changes at least in the web interface and popular
clients before pushing them to all users.

* Fix JavaScript interface with long IDs

Somewhat predictably, the JS interface handled IDs as numbers, which in
JS are IEEE double-precision floats. This loses some precision when
working with numbers as large as those generated by the new ID scheme,
so we instead handle them here as strings. This is relatively simple,
and doesn't appear to have caused any problems, but should definitely
be tested more thoroughly than the built-in tests. Several days of use
appear to support this working properly.

BREAKING CHANGE:

The major(!) change here is that IDs are now returned as strings by the
REST endpoints, rather than as integers. In practice, relatively few
changes were required to make the existing JS UI work with this change,
but it will likely hit API clients pretty hard: it's an entirely
different type to consume. (The one API client I tested, Tusky, handles
this with no problems, however.)

Twitter ran into this issue when introducing Snowflake IDs, and decided
to instead introduce an `id_str` field in JSON responses. I have opted
to *not* do that, and instead force all IDs to 64-bit integers
represented by strings in one go. (I believe Twitter exacerbated their
problem by rolling out the changes three times: once for statuses, once
for DMs, and once for user IDs, as well as by leaving an integer ID
value in JSON. As they said, "If you’re using the `id` field with JSON
in a Javascript-related language, there is a very high likelihood that
the integers will be silently munged by Javascript interpreters. In most
cases, this will result in behavior such as being unable to load or
delete a specific direct message, because the ID you're sending to the
API is different than the actual identifier associated with the
message." [1]) However, given that this is a significant change for API
users, alternatives or a transition time may be appropriate.

1: https://blog.twitter.com/developer/en_us/a/2011/direct-messages-going-snowflake-on-sep-30-2011.html

* Restructure feed pushes/unpushes

This was necessary because the previous behavior used Redis zset scores
to identify statuses, but those are IEEE double-precision floats, so we
can't actually use them to identify all 64-bit IDs. However, it leaves
the code in a much better state for refactoring reblog handling /
coalescing.

Feed-management code has been consolidated in FeedManager, including:

* BatchedRemoveStatusService no longer directly manipulates feed zsets
* RemoveStatusService no longer directly manipulates feed zsets
* PrecomputeFeedService has moved its logic to FeedManager#populate_feed

(PrecomputeFeedService largely made lots of calls to FeedManager, but
didn't follow the normal adding-to-feed process.)

This has the effect of unifying all of the feed push/unpush logic in
FeedManager, making it much more tractable to update it in the future.

Due to some additional checks that must be made during, for example,
batch status removals, some Redis pipelining has been removed. It does
not appear that this should cause significantly increased load, but if
necessary, some optimizations are possible in batch cases. These were
omitted in the pursuit of simplicity, but a batch_push and batch_unpush
would be possible in the future.

Tests were added to verify that pushes happen under expected conditions,
and to verify reblog behavior (both on pushing and unpushing). In the
case of unpushing, this includes testing behavior that currently leads
to confusion such as Mastodon's #2817, but this codifies that the
behavior is currently expected.

* Rubocop fixes

I could swear I made these changes already, but I must have lost them
somewhere along the line.

* Address review comments

This addresses the first two comments from review of this feature:

https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336735
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336931

This adds an optional argument to FeedManager#key, the subtype of feed
key to generate. It also tests to ensure that FeedManager's settings are
such that reblogs won't be tracked forever.

* Hardcode IdToBigints migration columns

This addresses a comment during review:
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139337452

This means we'll need to make sure that all _id columns going forward
are bigints, but that should happen automatically in most cases.

* Additional fixes for stringified IDs in JSON

These should be the last two. These were identified using eslint to try
to identify any plain casts to JavaScript numbers. (Some such casts are
legitimate, but these were not.)

Adding the following to .eslintrc.yml will identify casts to numbers:

~~~
  no-restricted-syntax:
  - warn
  - selector: UnaryExpression[operator='+'] > :not(Literal)
    message: Avoid the use of unary +
  - selector: CallExpression[callee.name='Number']
    message: Casting with Number() may coerce string IDs to numbers
~~~

The remaining three casts appear legitimate: two casts to array indices,
one in a server to turn an environment variable into a number.

* Only implement timestamp IDs for Status IDs

Per discussion in #4801, this is only being merged in for Status IDs at
this point. We do this in a migration, as there is no longer use for
a post-migration hook. We keep the initialization of the timestamp_id
function as a Rake task, as it is also needed after db:schema:load (as
db/schema.rb doesn't store Postgres functions).

* Change internal streaming payloads to stringified IDs as well

This is equivalent to 591a9af356faf2d5c7e66e3ec715502796c875cd from
#5019, with an extra change for the addition to FeedManager#unpush.

* Ensure we have a status_id_seq sequence

Apparently this is not a given when specifying a custom ID function,
so now we ensure it gets created. This uses the generic version of this
function to more easily support adding additional tables with timestamp
IDs in the future, although it would be possible to cut this down to a
less generic version if necessary. It is only run during db:schema:load
or the relevant migration, so the overhead is extraordinarily minimal.

* Transition reblogs to new Redis format

This provides a one-way migration to transition old Redis reblog entries
into the new format, with a separate tracking entry for reblogs.

It is not invertible because doing so could (if timestamp IDs are used)
require a database query for each status in each users' feed, which is
likely to be a significant toll on major instances.

* Address review comments from @akihikodaki

No functional changes.

* Additional review changes

* Heredoc cleanup

* Run db:schema:load hooks for test in development

This matches the behavior in Rails'
ActiveRecord::Tasks::DatabaseTasks.each_current_configuration, which
would otherwise break `rake db:setup` in development.

It also moves some functionality out to a library, which will be a good
place to put additional related functionality in the near future.
2017-10-04 09:56:37 +02:00
Yamagishi Kazutoshi d40c9140e8 Fix undefined local variable (regression from #5114) (#5210) 2017-10-04 09:51:42 +02:00
Eugen Rochko 291feba6f1 Follow up to #5172, clean up notifications after mute like after block (#5198) 2017-10-04 01:22:33 +02:00
Akihiko Odaki 63f0979799 Validate id of ActivityPub representations (#5114)
Additionally, ActivityPub::FetchRemoteStatusService no longer parses
activities.
OStatus::Activity::Creation no longer delegates to ActivityPub because
the provided ActivityPub representations are not signed while OStatus
representations are.
2017-10-04 01:13:48 +02:00
Eugen Rochko c743b5e1fd Fix possible acct: uri usurpation in ActivityPub account discovery (#5208)
Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>
2017-10-04 00:33:56 +02:00
Eugen Rochko 76f360c625 If HTTP signature is wrong and webfinger cache is stale, retry with resolve (#5129)
If the signature could not be verified and the webfinger of the account
was last retrieved longer than the cache period, try re-resolving the
account and then attempting to verify the signature again
2017-09-28 17:50:14 +02:00
abcang 3d9b8847d2 Flush body when POST requests (#5128) 2017-09-28 15:04:32 +02:00
Ondřej Hruška 83bda6c1a8
Merge commit 'b9f59ebcc68e9da0a7158741a1a2ef3564e1321e' into merging-upstream 2017-09-28 09:18:35 +02:00
Ondřej Hruška 9330ea1f4d
Merge commit '4aea3f88a6d30f102a79c2da7fcfac96465ba1a8' into merging-upstream 2017-09-28 09:12:17 +02:00
Eugen Rochko e528114c53 Follow-up to #4582 and #5027, removing dead code (#5101) 2017-09-26 01:06:27 +02:00
Eugen Rochko cf7fbf2c56 Fix #5059 - Stop processing payload if it's from local account (#5100) 2017-09-26 01:06:13 +02:00
Eugen Rochko 91e5b0dfdb Send streaming API delete to people mentioned in status (#5103)
- Previously they wouldn't receive it unless they were author's
  followers
- Skip unpush from public/hashtag timelines if status wasn't
  public in the first place
2017-09-26 00:29:29 +02:00
aschmitz 669fe9ee06 Change IDs to strings rather than numbers in API JSON output (#5019)
* Fix JavaScript interface with long IDs

Somewhat predictably, the JS interface handled IDs as numbers, which in
JS are IEEE double-precision floats. This loses some precision when
working with numbers as large as those generated by the new ID scheme,
so we instead handle them here as strings. This is relatively simple,
and doesn't appear to have caused any problems, but should definitely
be tested more thoroughly than the built-in tests. Several days of use
appear to support this working properly.

BREAKING CHANGE:

The major(!) change here is that IDs are now returned as strings by the
REST endpoints, rather than as integers. In practice, relatively few
changes were required to make the existing JS UI work with this change,
but it will likely hit API clients pretty hard: it's an entirely
different type to consume. (The one API client I tested, Tusky, handles
this with no problems, however.)

Twitter ran into this issue when introducing Snowflake IDs, and decided
to instead introduce an `id_str` field in JSON responses. I have opted
to *not* do that, and instead force all IDs to 64-bit integers
represented by strings in one go. (I believe Twitter exacerbated their
problem by rolling out the changes three times: once for statuses, once
for DMs, and once for user IDs, as well as by leaving an integer ID
value in JSON. As they said, "If you’re using the `id` field with JSON
in a Javascript-related language, there is a very high likelihood that
the integers will be silently munged by Javascript interpreters. In most
cases, this will result in behavior such as being unable to load or
delete a specific direct message, because the ID you're sending to the
API is different than the actual identifier associated with the
message." [1]) However, given that this is a significant change for API
users, alternatives or a transition time may be appropriate.

1: https://blog.twitter.com/developer/en_us/a/2011/direct-messages-going-snowflake-on-sep-30-2011.html

* Additional fixes for stringified IDs in JSON

These should be the last two. These were identified using eslint to try
to identify any plain casts to JavaScript numbers. (Some such casts are
legitimate, but these were not.)

Adding the following to .eslintrc.yml will identify casts to numbers:

~~~
  no-restricted-syntax:
  - warn
  - selector: UnaryExpression[operator='+'] > :not(Literal)
    message: Avoid the use of unary +
  - selector: CallExpression[callee.name='Number']
    message: Casting with Number() may coerce string IDs to numbers
~~~

The remaining three casts appear legitimate: two casts to array indices,
one in a server to turn an environment variable into a number.

* Back out RelationshipsController Change

This was made to make a test a bit less flakey, but has nothing to
do with this branch.

* Change internal streaming payloads to stringified IDs as well

Per
https://github.com/tootsuite/mastodon/pull/5019#issuecomment-330736452
we need these changes to send deleted status IDs as strings, not
integers.
2017-09-20 14:53:48 +02:00
Akihiko Odaki bb4d005a83 Introduce OStatus::TagManager (#5008) 2017-09-19 18:08:08 +02:00
Eugen Rochko 41e6c8b151 Fix incomplete account records being read (#4998)
* Fix incomplete account records being read

- Put account processing into redis lock
- Do not save until record is complete

* Fix spaces
2017-09-19 06:53:16 +02:00
Eugen Rochko 09a94b575e Admin interface for listing, adding and removing custom emojis (#5002)
* Admin interface for listing, adding and removing custom emojis

* Only display local ones in the list
2017-09-19 03:52:38 +02:00
unarist 1eab53ee10 Fix an error when actor json couldn't be fetched in ResolveRemoteAccountService (#4979)
* Fix an error when actor json couldn't be fetched in ResolveRemoteAccountService

* Add specs
2017-09-17 11:54:23 +02:00
Akihiko Odaki 48d77ea1eb Fix filterable_languages method of SettingsHelper (#4966) 2017-09-16 14:59:41 +02:00
ふぁぼ原 3816943e6b Enable to recognize most kinds of characters as URL paths (#4941) 2017-09-14 18:03:20 +02:00
Surinna Curtis f9d7b8a94f Refactor handling of default params for muting to make code cleaner 2017-09-13 21:47:30 -05:00
Surinna Curtis 0c547faf92 Less gross passing of notifications flag 2017-09-13 21:47:30 -05:00
Surinna Curtis 6f7d00bfdd Add support for muting notifications in MuteService 2017-09-13 21:47:30 -05:00
Surinna Curtis 27f8d7069b block notifications in notify_service from hard muted accounts 2017-09-13 21:47:30 -05:00
Eugen Rochko 596dab06e9 Support OpenGraph video embeds (#4897)
* Support OpenGraph video embeds

It's not really OpenGraph, it's twitter:player property, but it's
not OEmbed so that fits. For example, this allows Twitch clips to
be displayed as embeds.

Also, fixes glitch-soc/mastodon#135

* Fix invalid OpenGraph cards being saved through attaching and
revisit URLs after 14 days
2017-09-14 04:11:36 +02:00
ThibG af00220d79 Fix refollowing (#4931)
* Make RefollowWorker ActivityPub-only to avoid potential identifier mismatches

* Don't call RefollowWorker on new accounts
2017-09-14 00:05:25 +02:00
Eugen Rochko b9d241c6f5 Fix #4917 - Add missing suspend checks (#4921) 2017-09-13 11:05:02 +02:00
ThibG f29918e707 [WiP] Whenever a remote keypair changes, unfollow them and re-subscribe to … (#4907)
* Whenever a remote keypair changes, unfollow them and re-subscribe to them

In Mastodon (it could be different for other OStatus or AP-enabled software),
a keypair change is indicative of whole user (or instance) data loss. In this
situation, the “new” user might be different, and almost certainly has an empty
followers list. In this case, Mastodon instances will disagree on follower
lists, leading to unreliable delivery and “shadow followers”, that is users
believed by a remote instance to be followers, without the affected user
knowing.

Drawbacks of this change are:
1. If an user legitimately changes public key for some reason without losing
   data (not possible in Mastodon at the moment), they will have their remote
   followers unsubscribed/re-subscribed needlessly.
2. Depending of the number of remote followers, this may generate quite some
   traffic.
3. If the user change is an attempt at usurpation, the remote followers will
   unknowingly follow the usurper. Note that this is *not* a change of
   behavior, Mastodon already behaves like that, although delivery might be
   unreliable, and the usurper would not have known the former user's
   followers.

* Rename ResubscribeWorker to RefollowWorker

* Process followers in batches
2017-09-12 23:10:40 +02:00
David Yip 932571fa22 Merge tag 'v1.6.0' into sync/upstream 2017-09-10 13:04:27 -05:00
Eugen Rochko 7d7844a47f Default follows for new users (#4871)
When a new user confirms their e-mail, bootstrap their home timeline
by automatically following a set of accounts. By default, all local
admin accounts (that are unlocked). Can be customized by new admin
setting (comma-separated usernames, local and unlocked only)
2017-09-10 09:58:38 +02:00
David Yip 67ad453373 Merge tag 'v1.6.0rc4' into sync/upstream-1.6.0rc4
Conflicts:
      app/javascript/mastodon/features/getting_started/index.js
      app/javascript/packs/public.js
      app/javascript/styles/components.scss
2017-09-09 23:56:21 -05:00
David Yip 514fc908a3 Merge tag 'v1.6.0rc3' into sync/upstream 2017-09-09 14:28:08 -05:00
David Yip b9f7bc149b Merge branch 'origin/master' into sync/upstream
Conflicts:
	app/javascript/mastodon/components/status_list.js
	app/javascript/mastodon/features/notifications/index.js
	app/javascript/mastodon/features/ui/components/modal_root.js
	app/javascript/mastodon/features/ui/components/onboarding_modal.js
	app/javascript/mastodon/features/ui/index.js
	app/javascript/styles/about.scss
	app/javascript/styles/accounts.scss
	app/javascript/styles/components.scss
	app/presenters/instance_presenter.rb
	app/services/post_status_service.rb
	app/services/reblog_service.rb
	app/views/about/more.html.haml
	app/views/about/show.html.haml
	app/views/accounts/_header.html.haml
	config/webpack/loaders/babel.js
	spec/controllers/api/v1/accounts/credentials_controller_spec.rb
2017-09-09 14:27:47 -05:00
Eugen Rochko 90712d4293 Fix errors preventing UnsubscribeService from working (#4866) 2017-09-09 17:36:27 +02:00
Eugen Rochko 9e15eeec63 Add missing reject_media check before avatar download via ActivityPub (#4862) 2017-09-09 13:41:45 +02:00
Eugen Rochko 4b460bc571 Fix #4852 - Check if already requested from FollowService (#4855) 2017-09-09 02:02:44 +02:00
Eugen Rochko 7ca173be47 Fix #4850 - When visibility missing from API call to toot, fallback to user preference (#4861) 2017-09-09 02:02:29 +02:00
Eugen Rochko a4caa7eb62 Fetch statuses/following/followers numbers from ActivityPub collections (#4840) 2017-09-08 12:00:17 +02:00
Eugen Rochko e821c00e74 Fix mentions in direct statuses not being delivered via AP (#4806) 2017-09-05 20:55:25 +02:00
Eugen Rochko 9b50a9dd83 Fix some ActivityPub JSON bugs (#4796)
- Fix assumption that `url` is always a string. Handle it if it's an
  array of strings, array of objects, object, or string, both for
  accounts and for objects
- `sharedInbox` is actually supposed to be under `endpoints`, handle
  both cases and adjust the serializer
2017-09-04 18:26:33 +02:00
Eugen Rochko 37fdddd927 Rename "locked" to "manuallyApprovesFollowers" in ActivityPub (#4779)
See: <https://www.w3.org/wiki/Activity_Streams_extensions#as:manuallyApprovesFollowers>
2017-09-02 23:13:35 +02:00
Eugen Rochko d3b6746173 Make "unfollow" undo pending outgoing follow request too (#4781)
* Make "unfollow" undo pending outgoing follow request too

* Add cancel button to web UI when awaiting follow request approval

* Make the hourglass button do the cancelling
2017-09-02 20:44:41 +02:00
Eugen Rochko 1b5806b744 Define missing JSON-LD properties (#4767)
Using _: property names is discouraged, as in the future,
canonicalization may throw an error when encountering that instead
of discarding it silently like it does now.

We are defining some ActivityStreams properties which we expect
to land in ActivityStreams eventually, to ensure that future versions
of Mastodon will remain compatible with this even once that happens.
Those would be `locked`, `sensitive` and `Hashtag`

We are defining a custom context inline for some properties which we
do not expect to land in any other context. `atomUri`, `inReplyToAtomUri`
and `conversation` are part of the custom defined OStatus context.
2017-09-02 14:01:23 +02:00
Eugen Rochko 390bfec6da Avoid sending some ActivityPub payloads if the receiver will get them through distribution (#4739) 2017-09-01 21:26:01 +02:00
Eugen Rochko 7dc5035031 Make PreviewCard records reuseable between statuses (#4642)
* Make PreviewCard records reuseable between statuses

**Warning!** Migration truncates preview_cards tablec

* Allow a wider thumbnail for link preview, display it in horizontal layout (#4648)

* Delete preview cards files before truncating

* Rename old table instead of truncating it

* Add mastodon:maintenance:remove_deprecated_preview_cards

* Ignore deprecated_preview_cards in schema definition

* Fix null behaviour
2017-09-01 16:20:16 +02:00
unarist f7937d903c Don't process ActivityPub payload if signature is invalid (#4752)
* Don't process ActivityPub payload if signature is invalid

* Fix style issue
2017-08-31 17:18:49 +02:00
James 6b2be5dbfb Guarantee Subscription service first account has proper URL details (#4732)
* Guarantee Subscription service first account has proper URL details

Subscription Service potentially could break if the first user suspended
themselves, creating a situation where the urls that populate throughout
subscription service's PuSH request would cause the remote API to throw 503 errors.

Guaranteeing that the first account picked is not suspended prevents this problem.

* Fix style issue
2017-08-31 15:44:00 +02:00
Eugen Rochko 7b8f262840 Forward ActivityPub creates that reply to local statuses (#4709)
* Forward ActivityPub creates that reply to local statuses

* Fix test

* Fix wrong signers
2017-08-30 15:37:02 +02:00
Eugen Rochko 4c76402ba1 Serialize ActivityPub alternate link into OStatus deletes, handle it (#4730)
Requires moving Atom rendering from DistributionWorker (where
`stream_entry.status` is already nil) to inline (where
`stream_entry.status.destroyed?` is true) and distributing that.

Unfortunately, such XML renderings can no longer be easily chained
together into one payload of n items.
2017-08-29 16:11:05 +02:00
unarist 7876aed134 Fix deletion of status which has been reblogged (#4728) 2017-08-28 21:38:59 +02:00
Eugen Rochko 2a2698e450 Add ActivityPub serializer for Undo of Announce (#4703) 2017-08-26 15:32:40 +02:00
Eugen Rochko 00840f4f2e Add handling of Linked Data Signatures in payloads (#4687)
* Add handling of Linked Data Signatures in payloads

* Add a way to sign JSON, fix canonicalization of signature options

* Fix signatureValue encoding, send out signed JSON when distributing

* Add missing security context
2017-08-26 13:47:38 +02:00
masarakki 649a20ab46 authorize-follow-requests-after-unlocking (#4658) 2017-08-26 12:40:03 +02:00
Eugen Rochko cf615abbf9 Add configuration to disable private status federation over PuSH (#4582) 2017-08-24 17:51:32 +02:00