Commit Graph

857 Commits (3ab6a8b8bed37d048909b178693af8411325eb38)

Author SHA1 Message Date
David Yip a057ed5cfe
Merge remote-tracking branch 'tootsuite/master' into merge-upstream 2017-12-12 02:54:13 -06:00
Eugen Rochko 1356ed72cd
Fix #5953 - Add GET /api/v1/accounts/:id/lists (#5983) 2017-12-12 03:55:39 +01:00
Akihiko Odaki e56323a4dd Remove preview_card fabricator (#5975)
preview_card fabricator has a removed attribute, status, and is no longer
functional.
2017-12-11 22:22:08 +09:00
Eugen Rochko a865b62efc
Rate limit by user instead of IP when API user is authenticated (#5923)
* Fix #668 - Rate limit by user instead of IP when API user is authenticated

* Fix code style issue

* Use request decorator provided by Doorkeeper
2017-12-09 14:20:02 +01:00
abcang 99242b92bc Keep WebPush settings (#5879) 2017-12-09 02:31:37 +01:00
Jenkins 8ca91cef45 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-12-06 20:17:13 +00:00
aschmitz 4de211b80a Break out nested relationship API keys (#5887)
* Break out nested relationship API keys

This closes #5856 by restoring the existing behavior of the `muting`
and `following` keys (returning booleans rather than truthy or false).
It adds `showing_reblogs` and `muting_notifications` keys:

* `showing_reblogs` returns true if:
  1. You've requested to follow the user, with reblogs shown, or
  2. You are following the user, with reblogs shown.
* `muting_notifications` returns true if you have muted the user and
  their notifications as well.

* Rubocop fix

* Fix pulling reblog/mute status from relationships

I could swear this had passed tests before, but apparently not.
Works now.

* More test fixes

Really, you'd expect this to be more straightforward.
2017-12-06 16:10:54 +01:00
Yamagishi Kazutoshi b21db9bbde Using double splat operator (#5859) 2017-12-06 11:41:57 +01:00
Eugen Rochko e20895f251
Add list of lists component to web UI (#5811)
* Add list of lists component to web UI

* Add list adding

* Add list removing

* List editor modal

* Add API account search limited by following=true relation

* Rework list editor modal

* Remove mandatory pagination of GET /api/v1/lists/:id/accounts

* Adjust search input placeholder

* Fix rspec (#5890)

* i18n: (zh-CN) Add missing translations for #5811 (#5891)

* i18n: (zh-CN) yarn manage:translations -- zh-CN

* i18n: (zh-CN) Add missing translations for #5811

* Fix some issues

- Display loading/missing state for list timelines
- Order lists alphabetically in overview
- Fix async list editor reset
- Redirect to /lists after deleting unpinned list
- Redirect to / after pinning a list

* Remove dead list columns when a list is deleted or fetch returns 404
2017-12-05 23:02:27 +01:00
Jenkins ad46bc9772 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-30 03:17:12 +00:00
Eugen Rochko 4c6b5dbe96
Add semi-support for Video/Image objects in ActivityPub (#5848)
* Add semi-support for Video/Image objects in ActivityPub

Video and Image objects will create corresponding status records
with manually crafted text contents (title + URL)

* Extract html-url-finding logic into JsonLdHelper

* Fallback to id when url missing, extract supported object types
2017-11-30 04:06:20 +01:00
Eugen Rochko 85e97ecab6
Fix too many forwards (#5854)
* Avoid sending explicit Undo->Announce when original deleted

* Do not forward a reply back to the server that sent it

* Deduplicate inboxes of rebloggers' followers for delete forwarding

* Adjust test

* Fix wrong class, bad SQL, wrong variable, outdated comment
2017-11-30 03:50:05 +01:00
David Yip 95c270f5b1
Merge remote-tracking branch 'origin/master' into gs-master 2017-11-28 11:45:13 -06:00
aschmitz eeaec39888 Allow hiding of reblogs from followed users (#5762)
* Allow hiding of reblogs from followed users

This adds a new entry to the account menu to allow users to hide
future reblogs from a user (and then if they've done that, to show
future reblogs instead).

This does not remove or add historical reblogs from/to the user's
timeline; it only affects new statuses.

The API for this operates by sending a "reblogs" key to the follow
endpoint. If this is sent when starting a new follow, it will be
respected from the beginning of the follow relationship (even if
the follow request must be approved by the followee). If this is
sent when a follow relationship already exists, it will simply
update the existing follow relationship. As with the notification
muting, this will now return an object ({reblogs: [true|false]}) or
false for each follow relationship when requesting relationship
information for an account. This should cause few issues due to an
object being truthy in many languages, but some modifications may
need to be made in pickier languages.

Database changes: adds a show_reblogs column (default true,
non-nullable) to the follows and follow_requests tables. Because
these are non-nullable, we use the existing MigrationHelpers to
perform this change without locking those tables, although the
tables are likely to be small anyway.

Tests included.

See also <https://github.com/glitch-soc/mastodon/pull/212>.

* Rubocop fixes

* Code review changes

* Test fixes

This patchset closes #648 and resolves #3271.

* Rubocop fix

* Revert reblogs defaulting in argument, fix tests

It turns out we needed this for the same reason we needed it in muting:
if nil gets passed in somehow (most usually by an API client not passing
any value), we need to detect and handle it.

We could specify a default in the parameter and then also catch nil, but
there's no great reason to duplicate the default value.
2017-11-28 15:00:35 +01:00
ThibG b8efb5daed Fix handling of temporary failures in ProcessMentionsService (#5842)
* Add test for temporary account resolving failures in ProcessMentionsService

* Fix processing of mentions to already-known remote accounts on temporary failures
2017-11-28 15:00:22 +01:00
David Yip e77c3996a5
Merge remote-tracking branch 'origin/master' into gs-master 2017-11-27 11:29:04 -06:00
Eugen Rochko 740f8a95a9
Add consumable invites (#5814)
* Add consumable invites

* Add UI for generating invite codes

* Add tests

* Display max uses and expiration in invites table, delete invite

* Remove unused column and redundant validator

- Default follows not used, probably bad idea
- InviteCodeValidator is redundant because RegistrationsController
  checks invite code validity

* Add admin setting to disable invites

* Add admin UI for invites, configurable role for invite creation

- Admin UI that lists everyone's invites, always available
- Admin setting min_invite_role to control who can invite people
- Non-admin invite UI only visible if users are allowed to

* Do not remove invites from database, expire them instantly
2017-11-27 16:07:59 +01:00
Jenkins 86f4f8e158 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-25 05:17:15 +00:00
ysksn 1104ac35d3 Add tests for Streamable (#5771) 2017-11-24 14:42:57 +01:00
ysksn 8c0e77d688 Add tests for AccountRelationshipsPresenter (#5805) 2017-11-24 14:41:04 +01:00
Jenkins 167fe2ab08 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-24 01:17:20 +00:00
Eugen Rochko e84fecb7e9
Add logging of admin actions (#5757)
* Add logging of admin actions

* Update brakeman whitelist

* Log creates, updates and destroys with history of changes

* i18n: Update Polish translation (#5782)

Signed-off-by: Marcin Mikołajczak <me@m4sk.in>

* Split admin navigation into moderation and administration

* Redesign audit log page

* 🇵🇱 (#5795)

* Add color coding to audit log

* Change dismiss->resolve, log all outcomes of report as resolve

* Update terminology (e-mail blacklist) (#5796)

* Update terminology (e-mail blacklist)

imho looks better

* Update en.yml

* Fix code style issues

* i18n-tasks normalize
2017-11-24 02:05:53 +01:00
Jenkins 8f3e5f6128 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-21 05:17:12 +00:00
ysksn 6f609dc4b4 Add tests for Remotable (#5768) 2017-11-21 13:59:36 +09:00
Jenkins a36a2c1796 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-19 15:17:12 +00:00
abcang 53e95c4efc Fix N+1 at notification (#5752) 2017-11-19 15:32:48 +01:00
David Yip e45cb0837b Update .following_map examples with show-reblogs information. 2017-11-19 02:56:45 -06:00
David Yip d083f7741a Merge remote-tracking branch 'tootsuite/master' 2017-11-19 01:30:29 -06:00
ysksn 2590aac863 Add tests for AccountInteractions (#5751)
* Add tests for class methods of AccountInteractions

* Add tests for instance methods of AccountInteractions
2017-11-19 04:16:14 +01:00
ysksn 8f638a2bf2 Add a test for Settings::Extend#settings (#5720) 2017-11-19 04:15:32 +01:00
ysksn 9d9b1aff1e Add tests for Status#title (#5718) 2017-11-19 04:15:17 +01:00
beatrix dec960c828
Merge pull request #208 from yipdw/keyword-mute
Run keyword mutes on hashtags
2017-11-18 12:06:40 -05:00
David Yip 1ab12ba38e Merge remote-tracking branch 'origin/master' into merge-upstream 2017-11-17 17:40:00 -06:00
Eugen Rochko 24cafd73a2
Lists (#5703)
* Add structure for lists

* Add list timeline streaming API

* Add list APIs, bind list-account relation to follow relation

* Add API for adding/removing accounts from lists

* Add pagination to lists API

* Add pagination to list accounts API

* Adjust scopes for new APIs

- Creating and modifying lists merely requires "write" scope
- Fetching information about lists merely requires "read" scope

* Add test for wrong user context on list timeline

* Clean up tests
2017-11-18 00:16:48 +01:00
David Yip f6355f6ffb Update StatusPolicy to check current_account for local_only? toots.
StatusPolicy#account was renamed to StatusPolicy#current_account in
upstream.  This commit renames the local-only changes to match and
augments the #show? policy spec with what we expect for local-only
toots.
2017-11-17 09:07:21 -06:00
David Yip b28b405b97 Merge remote-tracking branch 'origin/master' into gs-master 2017-11-16 22:52:37 -06:00
ysksn 9dd5e329ab Remove empty strings (#5732) 2017-11-17 10:52:30 +09:00
Surinna Curtis 35fbdc36f9 Merge tootsuite/master at 3023725936 2017-11-16 01:21:16 -06:00
ysksn 556c07df1f Add tests for Status#verb (#5717) 2017-11-16 15:46:29 +09:00
David Yip c2a92dffc9 Add some examples for Glitch::KeywordMute::TagMatcher. #208. 2017-11-15 23:31:49 -06:00
ysksn 3023725936 Add tests for Status#hidden? (#5719) 2017-11-16 11:07:27 +09:00
David Yip 08652baab0 Replace =~ with #matches?. #208.
=~ made sense when we were passing it through to a regex, but we're no
longer doing that: TagMatcher looks at individual tags and returns a
value that *looks* like what you get out of #=~ but really isn't that
meaningful.  Probably a good idea to not subvert convention like this
and instead use a name with guessable intent.
2017-11-15 18:27:25 -06:00
David Yip cb4ef24ac9 Match keyword mute filter on hashtags. #208.
It is reasonable to expect someone to enter #foo to mute hashtag #foo.
However, tags are recorded on statuses without the preceding #.
To adjust for this, we build a separate tag matcher and use
Tag::HASHTAG_RE to extract a hashtag from the hashtag syntax.
2017-11-15 18:12:29 -06:00
David Yip 431503bae2 Also run the keyword matcher on a status' tags. #208. 2017-11-15 18:12:26 -06:00
beatrix 04508868b0
Merge pull request #212 from aschmitz/feat/mute-reblogs
Allow hiding reblogs on a per-follow basis
2017-11-15 12:01:17 -05:00
ysksn 19e8b861a2 Delegate some methods of User to @settings (#5706)
* Move some tests of User into Settings::ScopedSettings

* Add a test for User@settings
2017-11-15 16:05:20 +01:00
ysksn 7d7df877ef Add a test for Tag#to_param (#5705) 2017-11-15 16:04:41 +01:00
Surinna Curtis 031a5a8f92 Optional notification muting (#5087)
* Add a hide_notifications column to mutes

* Add muting_notifications? and a notifications argument to mute!

* block notifications in notify_service from hard muted accounts

* Add specs for how mute! interacts with muting_notifications?

* specs testing that hide_notifications in mutes actually hides notifications

* Add support for muting notifications in MuteService

* API support for muting notifications (and specs)

* Less gross passing of notifications flag

* Break out a separate mute modal with a hide-notifications checkbox.

* Convert profile header mute to use mute modal

* Satisfy eslint.

* specs for MuteService notifications params

* add trailing newlines to files for Pork :)

* Put the label for the hide notifications checkbox in a label element.

* Add a /api/v1/mutes/details route that just returns the array of mutes.

* Define a serializer for /api/v1/mutes/details

* Add more specs for the /api/v1/mutes/details endpoint

* Expose whether a mute hides notifications in the api/v1/relationships endpoint

* Show whether muted users' notifications are muted in account lists

* Allow modifying the hide_notifications of a mute with the /api/v1/accounts/:id/mute endpoint

* make the hide/unhide notifications buttons work

* satisfy eslint

* In probably dead code, replace a dispatch of muteAccount that was skipping the modal with launching the mute modal.

* fix a missing import

* add an explanatory comment to AccountInteractions

* Refactor handling of default params for muting to make code cleaner

* minor code style fixes oops

* Fixed a typo that was breaking the account mute API endpoint

* Apply white-space: nowrap to account relationships icons

* Fix code style issues

* Remove superfluous blank line

* Rename /api/v1/mutes/details -> /api/v2/mutes

* Don't serialize "account" in MuteSerializer

Doing so is somewhat unnecessary since it's always the current user's account.

* Fix wrong variable name in api/v2/mutes

* Use Toggle in place of checkbox in the mute modal.

* Make the Toggle in the mute modal look better

* Code style changes in specs and removed an extra space

* Code review suggestions from akihikodaki

Also fixed a syntax error in tests for AccountInteractions.

* Make AddHideNotificationsToMute Concurrent

It's not clear how much this will benefit instances in practice, as the
number of mutes tends to be pretty small, but this should prevent any
blocking migrations nonetheless.

* Fix up migration things

* Remove /api/v2/mutes
2017-11-15 03:56:41 +01:00
ysksn 6d7e05ec1f Add tests for StreamEntry (#5687)
* Add tests for StreamEntry

- `#object_type`
- `#verb`
- `#mentions`

* Fix to test results instead of implementations
2017-11-15 02:00:58 +01:00
Eugen Rochko fbef909c2a
Add option to block direct messages from people you don't follow (#5669)
* Add option to block direct messages from people you don't follow

Fix #5326

* If the DM responds to a toot by recipient, allow it through

* i18n: Update Polish translation (for #5669) (#5673)
2017-11-14 21:12:57 +01:00
ysksn 48e27c47a7 Add a test for SiteUpload#cache_key (#5685) 2017-11-14 20:44:11 +01:00
Yamagishi Kazutoshi 1f1838420f Refactor remote_follow_spec.rb (#5690) 2017-11-14 20:41:17 +01:00
Yamagishi Kazutoshi 20150659e6 Add uniqueness to block email domains (#5692) 2017-11-14 20:37:17 +01:00
ysksn 4112a0631f Add tests for Setting (#5683) 2017-11-14 11:08:04 +09:00
David Yip 656d54e945 Maintain case-insensitivity when merging multiple matchers (#213)
When given two regexps, Regexp.union preserves the options set (or not
set) on each regex; this meant that none of the multiline (m),
case-insensitivity (i), or extended syntax (x) options were set.  Our
regexps are written expecting the m, i, and x options were set on all of
them, so we need to make sure that we preserve that behavior.
2017-11-13 11:06:02 -06:00
ysksn 60f247c2e7 Add tests for SessionActivation (#5668)
* Fabricate SessionActivation

not only user_id but user association.

* Add tests for SessionActivation
2017-11-13 09:54:48 +09:00
ysksn 2fb722397d Add tests for RemoteProfile (#5665) 2017-11-12 16:23:31 +09:00
aschmitz 9a42f7cbed Actually filter blocked reblogs from feed
And even a relevant test. Whoops.
2017-11-11 22:10:49 -06:00
aschmitz b95c48748c Per-user reblog hiding implementation/fixes/tests
Note that this will only hide/show *future* reblogs by a user, and does
nothing to remove/add reblogs that are already in the timeline. I don't
think that's a particularly confusing behavior, and it's a lot easier
to implement (similar to mutes, I believe).
2017-11-10 22:04:54 -06:00
ysksn 56720ba590 Add tests for RemoteFollow (#5651)
* Add tests for RemoteFollow.initialize

* Add tests for RemoteFollow#valid?

* Add tests for RemoteFollow#subscribe_address_for
2017-11-10 16:56:02 +01:00
ysksn 07cca6e364 Add tests for Notification (#5640)
* Add tests for Notification#target_status

* Add tests for Notification#browserable?

* Add tests for Notification.reload_stale_associations!
2017-11-09 14:37:10 +01:00
ysksn 54b42901df Add and Remove tests for FollowRequest (#5622)
* Add a test for FollowRequest#authorize!

* Remove tests

There is no need to test
ActiveModel::Validations::ClassMethods#validates.

* Make an alias of destroy! as reject!

Instead of defining the method,
make an alias of destroy! as reject! because of reducing test.
2017-11-09 14:36:52 +01:00
Nanamachi d200e041fe Rewrite account_controller_spec (#5633)
* make accounts_controller_spec DRY

* Add blocked user spec
2017-11-09 14:36:17 +01:00
ysksn 64cc129225 Add tests for MediaAttachment (#5620)
- `#local?`
- `#needs_redownload?`
- `#to_param`
2017-11-08 15:29:07 +09:00
ysksn 97fc2da2e0 Add tests for CustomEmoji#local? and #object_type (#5621) 2017-11-08 15:28:17 +09:00
ysksn d307ee79e9 Implement tests for Account#refresh! (#5601) 2017-11-06 13:54:41 +09:00
ysksn cf01326cc1 Add test for Account#save_with_optional_media! (#5603)
There was a test when some of the properties are invalid, but none when all
of them are valid.
2017-11-06 13:54:12 +09:00
ysksn 15c0f6ae56 Implement tests for Account#possibly_stale? (#5591) 2017-11-05 17:20:05 +09:00
David Yip dd28b94cf0 Merge remote-tracking branch 'origin/master' into gs-master 2017-10-26 09:18:27 -05:00
unarist 4f337c020a Fix Cocaine::ExitStatusError when upload small non-animated GIF (#5489)
Looks like copied tempfile need to be flushed before further processing. This issue won't happen if the uploaded file has enough file size.
2017-10-26 22:48:35 +09:00
beatrix a2612d0d38 Merge pull request #179 from glitch-soc/keyword-mute
Keyword muting
2017-10-25 17:37:48 -04:00
David Yip 42f2045c21 Merge remote-tracking branch 'STJrInuyasha/feature/direct-timeline' into gs-direct-timeline 2017-10-25 16:01:20 -05:00
David Yip d03b48cea0 Also filter notifications containing muted keywords. 2017-10-24 18:51:27 -05:00
David Yip 641f90e73a Fix example description.
This example actually checks matches at the end of a string.
2017-10-24 18:33:02 -05:00
Matthew Walsh 3db80f75a6 Added a timeline for Direct statuses
* Lists all Direct statuses you've sent and received
* Displayed in Getting Started
* Streaming server support for direct TL
2017-10-22 18:35:14 -07:00
David Yip 4b68e82a19 Don't add \b to whole-word keywords that don't start with word characters.
Ditto for ending with \b.

Consider muting the phrase "(hot take)".  I stipulate it is reasonable
to enter this with the default "match whole word" behavior.  Under the
old behavior, this would be encoded as

    \b\(hot\ take\)\b

However, if \b is before the first character in the string and the first
character in the string is not a word character, then the match will
fail.  Ditto for after.  In our example, "(" is not a word character, so
this will not match statuses containing "(hot take)", and that's a very
surprising behavior.

To address this, we only add leading and trailing \b to keywords that
start or end with word characters.
2017-10-22 00:38:54 -05:00
David Yip 19826774f0 keyword mutes: also check spoiler (CW) text and reblogged statuses. 2017-10-22 00:38:53 -05:00
David Yip ad86c86fa8 Apply keyword mutes to reblogs. 2017-10-21 15:44:47 -05:00
David Yip 670e6a33f8 Move KeywordMute into Glitch namespace.
There are two motivations for this:

1. It looks like we're going to add other features that require
   server-side storage (e.g. user notes).

2. Namespacing glitchsoc modifications is a good idea anyway: even if we
   do not end up doing (1), if upstream introduces a keyword-mute feature
   that also uses a "KeywordMute" model, we can avoid some merge
   conflicts this way and work on the more interesting task of
   choosing which implementation to use.
2017-10-21 14:54:36 -05:00
David Yip 4a64181461 Allow keywords to match either substrings or whole words.
Word-boundary matching only works as intended in English and languages
that use similar word-breaking characters; it doesn't work so well in
(say) Japanese, Chinese, or Thai.  It's unacceptable to have a feature
that doesn't work as intended for some languages.  (Moreso especially
considering that it's likely that the largest contingent on the Mastodon
bit of the fediverse speaks Japanese.)

There are rules specified in Unicode TR29[1] for word-breaking across
all languages supported by Unicode, but the rules deliberately do not
cover all cases.  In fact, TR29 states

    For example, reliable detection of word boundaries in languages such
    as Thai, Lao, Chinese, or Japanese requires the use of dictionary
    lookup, analogous to English hyphenation.

So we aren't going to be able to make word detection work with regexes
within Mastodon (or glitchsoc).  However, for a first pass (even if it's
kind of punting) we can allow the user to choose whether they want word
or substring detection and warn about the limitations of this
implementation in, say, docs.

[1]: https://unicode.org/reports/tr29/
     https://web.archive.org/web/20171001005125/https://unicode.org/reports/tr29/
2017-10-21 14:54:36 -05:00
David Yip 4fa2f7e82d Set up /settings/keyword_mutes. #164.
This should eventually be accessible via the API and the web frontend,
but I find it easier to set up an editing interface using Rails
templates and the like.  We can always take it out if it turns out we
don't need it.
2017-10-21 14:54:36 -05:00
David Yip 9f609bc94e Fix case-insensitive match scenario; test some word ornamentation. #164. 2017-10-21 14:54:36 -05:00
David Yip 603cf02b70 Rework KeywordMute interface to use a matcher object; spec out matcher. #164.
A matcher object that builds a match from KeywordMute data and runs it
over text is, in my view, one of the easier ways to write examples for
this sort of thing.
2017-10-21 14:54:36 -05:00
David Yip 4745d6eeca Spec out KeywordMute interface. #164. 2017-10-21 14:54:21 -05:00
David Yip 9093e2de7a Add KeywordMute model.
Gist of the proposed keyword mute implementation:

Keyword mutes are represented server-side as one keyword per record.
For each account, there exists a keyword regex that is generated as one
big alternation of all keywords.  This regex is cached (in Redis, I
guess) so we can quickly get it when filtering in FeedManager.
2017-10-21 14:53:41 -05:00
beatrix a7be86e875 hide mentions of muted accounts (in home col) (#190)
* hide mentions of muted accounts (in home col)

also cleans up some old crap

* add test
2017-10-20 10:49:54 -04:00
David Yip 42e8c8eb0e Merge tag 'v2.0.0' into gs-master 2017-10-18 11:52:04 -05:00
aschmitz 554c2fd8af Clean up reblog tracking keys, related improvements (#5428)
* Clean up reblog-tracking sets from FeedManager

Builds on #5419, with a few minor optimizations and cleanup of sets
after they are no longer needed.

* Update tests, fix multiply-reblogged case

Previously, we would have lost the fact that a given status was
reblogged if the displayed reblog of it was removed, now we don't.

Also added tests to make sure FeedManager#trim cleans up our reblog
tracking keys, fixed up FeedCleanupScheduler to use the right loop,
and fixed the test for it.
2017-10-17 11:45:06 +02:00
Ondřej Hruška 8b9ee5f16b
Merge remote-tracking branch 'upstream/master' into upstream-merge-again 2017-10-16 21:13:47 +02:00
Eugen Rochko 34118169ac Keep references to all reblogs of a status on home feed (#5419)
* Keep references to all reblogs of a status on home feed

When inserting reblog: Add to set of reblogs of this status on
the feed, if original status was present in the feed, add it to
that set as well.

When removing a reblog: Remove it from that set. Take random
remaining item from the set. If one exists, re-insert it into feed,
otherwise do not re-insert anything.

Fix #4210

* When original is removed, toss out reblog references
2017-10-16 20:44:31 +02:00
David Yip dbb1fce94d Merge remote-tracking branch 'upstream/master' into gs-master 2017-10-16 09:23:59 -05:00
unarist 6f490b4bfe Fix un-reblogged status being at wrong position in the home timeline (#5418)
We've changed un-reblogging behavior when we implement Snowflake, to insert un-reblogged status at the position reblogging status existed.

However, our API expects home timeline is ordered by status ids, and max_id/since_id filters by zset score. Due to this, un-reblogged status appears as a last item of result set, and timeline expansion may skips many statuses.

So this reverts that change...reblogged status inserted at corresponding position to its id.
2017-10-16 15:58:23 +02:00
Yamagishi Kazutoshi d5b767c374 Replace JavaScript Testing Framework from Mocha to Jest (#5412) 2017-10-16 09:33:08 +02:00
David Yip 6cd5b3bbe5 Merge remote-tracking branch 'origin/master' into gs-master 2017-10-16 01:29:02 -05:00
unarist 8125fdc19f Use atomUri in Undo activity of Announce (#5376)
This allows deletion of reblogs which delivered before with OStatus URI.
2017-10-14 14:42:09 +02:00
Eugen Rochko 3283868e28 Improve spec of Feed and UserTrackingConcern (#5367) 2017-10-13 11:00:11 +02:00
Lex Alexander b8bae96647 Retoot count increases without reason (#5363)
* Retoot count increases without reason

-The store_uri method for Statuses was being called on after_create and causing reblogs to be incremented twice.
-This calls it when the transaction is finished by using after_create_commit.
-Fixes #4916.

* Added test case for after_create_commit callback for checking reblog count.

* Rewrote test to keep original, but added one for only the after_create_commit callback.
2017-10-13 02:52:09 +02:00
kibigo! 8d6b9ba494 Merge upstream 2.0ish #165 2017-10-11 10:43:10 -07:00
Akihiko Odaki cc796298c9 Fix pagination in Api::V1::BlocksController (#5285) 2017-10-09 17:30:31 +02:00
Eugen Rochko 0717d9b3e6 Set snowflake IDs for backdated statuses (#5260)
- Rename Mastodon::TimestampIds into Mastodon::Snowflake for clarity
- Skip for statuses coming from inbox, aka delivered in real-time
- Skip for statuses that claim to be from the future
2017-10-08 17:34:34 +02:00
nullkal 633426b261 Add moderation note (#5240)
* Add moderation note

* Add frozen_string_literal

* Make rspec pass
2017-10-07 20:26:43 +02:00
Eugen Rochko f486ef2666 Redesign public hashtag pages (#5237) 2017-10-07 20:00:35 +02:00
Eugen Rochko 3a3475450e Encode custom emojis as resolveable objects in ActivityPub (#5243)
* Encode custom emojis as resolveable objects in ActivityPub

* Improve code style
2017-10-07 17:43:42 +02:00
Nolan Lawson 057db0ecd0 Update emoji-mart to v2.1.1 (#5256) 2017-10-07 12:17:02 +02:00
Nolan Lawson 72d939b69f Fix thinking_face emoji autocomplete (#5238) 2017-10-06 12:03:13 +02:00
Ondřej Hruška dcd8ff5308
fix haml quotes in a bid to make travis less grumpy 2017-10-06 09:07:04 +02:00
Nolan Lawson fd7f0732fe Compress and combine emoji data (#5229) 2017-10-06 03:42:34 +02:00
ThibG 2559d9166c Fix regression in FetchRemoteResourceService (#5217)
* Fix regression in FetchRemoteResourceService

* Update specs to match interface changes made in #5114
2017-10-05 00:21:44 +02:00
utam0k b3af3f9f8c Implement EmailBlackList (#5109)
* Implement BlacklistedEmailDomain

* Use Faker::Internet.domain_name

* Remove note column

* Add frozen_string_literal comment

* Delete unnecessary codes

* Sort alphabetically

* Change of wording

* Rename BlacklistedEmailDomain to EmailDomainBlock
2017-10-04 15:16:10 +02:00
Yamagishi Kazutoshi 178f718a9b Separate notifications preferences from general preferences (#4447)
* Separate notifications preferences from general preferences

* Refine settings/notifications/show

* remove preferences.notifications
2017-10-04 10:22:52 +02:00
aschmitz 468523f4ad Non-Serial ("Snowflake") IDs (#4801)
* Use non-serial IDs

This change makes a number of nontrivial tweaks to the data model in
Mastodon:

* All IDs are now 8 byte integers (rather than mixed 4- and 8-byte)
* IDs are now assigned as:
  * Top 6 bytes: millisecond-resolution time from epoch
  * Bottom 2 bytes: serial (within the millisecond) sequence number
  * See /lib/tasks/db.rake's `define_timestamp_id` for details, but
    note that the purpose of these changes is to make it difficult to
    determine the number of objects in a table from the ID of any
    object.
* The Redis sorted set used for the feed will have values used to look
  up toots, rather than scores. This is almost always the same as the
  existing behavior, except in the case of boosted toots. This change
  was made because Redis stores scores as double-precision floats,
  which cannot store the new ID format exactly. Note that this doesn't
  cause problems with sorting/pagination, because ZREVRANGEBYSCORE
  sorts lexicographically when scores are tied. (This will still cause
  sorting issues when the ID gains a new significant digit, but that's
  extraordinarily uncommon.)

Note a couple of tradeoffs have been made in this commit:

* lib/tasks/db.rake is used to enforce many/most column constraints,
  because this commit seems likely to take a while to bring upstream.
  Enforcing a post-migrate hook is an easier way to maintain the code
  in the interim.
* Boosted toots will appear in the timeline as many times as they have
  been boosted. This is a tradeoff due to the way the feed is saved in
  Redis at the moment, but will be handled by a future commit.

This would effectively close Mastodon's #1059, as it is a
snowflake-like system of generating IDs. However, given how involved
the changes were simply within Mastodon, it may have unexpected
interactions with some clients, if they store IDs as doubles
(or as 4-byte integers). This was a problem that Twitter ran into with
their "snowflake" transition, particularly in JavaScript clients that
treated IDs as JS integers, rather than strings. It therefore would be
useful to test these changes at least in the web interface and popular
clients before pushing them to all users.

* Fix JavaScript interface with long IDs

Somewhat predictably, the JS interface handled IDs as numbers, which in
JS are IEEE double-precision floats. This loses some precision when
working with numbers as large as those generated by the new ID scheme,
so we instead handle them here as strings. This is relatively simple,
and doesn't appear to have caused any problems, but should definitely
be tested more thoroughly than the built-in tests. Several days of use
appear to support this working properly.

BREAKING CHANGE:

The major(!) change here is that IDs are now returned as strings by the
REST endpoints, rather than as integers. In practice, relatively few
changes were required to make the existing JS UI work with this change,
but it will likely hit API clients pretty hard: it's an entirely
different type to consume. (The one API client I tested, Tusky, handles
this with no problems, however.)

Twitter ran into this issue when introducing Snowflake IDs, and decided
to instead introduce an `id_str` field in JSON responses. I have opted
to *not* do that, and instead force all IDs to 64-bit integers
represented by strings in one go. (I believe Twitter exacerbated their
problem by rolling out the changes three times: once for statuses, once
for DMs, and once for user IDs, as well as by leaving an integer ID
value in JSON. As they said, "If you’re using the `id` field with JSON
in a Javascript-related language, there is a very high likelihood that
the integers will be silently munged by Javascript interpreters. In most
cases, this will result in behavior such as being unable to load or
delete a specific direct message, because the ID you're sending to the
API is different than the actual identifier associated with the
message." [1]) However, given that this is a significant change for API
users, alternatives or a transition time may be appropriate.

1: https://blog.twitter.com/developer/en_us/a/2011/direct-messages-going-snowflake-on-sep-30-2011.html

* Restructure feed pushes/unpushes

This was necessary because the previous behavior used Redis zset scores
to identify statuses, but those are IEEE double-precision floats, so we
can't actually use them to identify all 64-bit IDs. However, it leaves
the code in a much better state for refactoring reblog handling /
coalescing.

Feed-management code has been consolidated in FeedManager, including:

* BatchedRemoveStatusService no longer directly manipulates feed zsets
* RemoveStatusService no longer directly manipulates feed zsets
* PrecomputeFeedService has moved its logic to FeedManager#populate_feed

(PrecomputeFeedService largely made lots of calls to FeedManager, but
didn't follow the normal adding-to-feed process.)

This has the effect of unifying all of the feed push/unpush logic in
FeedManager, making it much more tractable to update it in the future.

Due to some additional checks that must be made during, for example,
batch status removals, some Redis pipelining has been removed. It does
not appear that this should cause significantly increased load, but if
necessary, some optimizations are possible in batch cases. These were
omitted in the pursuit of simplicity, but a batch_push and batch_unpush
would be possible in the future.

Tests were added to verify that pushes happen under expected conditions,
and to verify reblog behavior (both on pushing and unpushing). In the
case of unpushing, this includes testing behavior that currently leads
to confusion such as Mastodon's #2817, but this codifies that the
behavior is currently expected.

* Rubocop fixes

I could swear I made these changes already, but I must have lost them
somewhere along the line.

* Address review comments

This addresses the first two comments from review of this feature:

https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336735
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336931

This adds an optional argument to FeedManager#key, the subtype of feed
key to generate. It also tests to ensure that FeedManager's settings are
such that reblogs won't be tracked forever.

* Hardcode IdToBigints migration columns

This addresses a comment during review:
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139337452

This means we'll need to make sure that all _id columns going forward
are bigints, but that should happen automatically in most cases.

* Additional fixes for stringified IDs in JSON

These should be the last two. These were identified using eslint to try
to identify any plain casts to JavaScript numbers. (Some such casts are
legitimate, but these were not.)

Adding the following to .eslintrc.yml will identify casts to numbers:

~~~
  no-restricted-syntax:
  - warn
  - selector: UnaryExpression[operator='+'] > :not(Literal)
    message: Avoid the use of unary +
  - selector: CallExpression[callee.name='Number']
    message: Casting with Number() may coerce string IDs to numbers
~~~

The remaining three casts appear legitimate: two casts to array indices,
one in a server to turn an environment variable into a number.

* Only implement timestamp IDs for Status IDs

Per discussion in #4801, this is only being merged in for Status IDs at
this point. We do this in a migration, as there is no longer use for
a post-migration hook. We keep the initialization of the timestamp_id
function as a Rake task, as it is also needed after db:schema:load (as
db/schema.rb doesn't store Postgres functions).

* Change internal streaming payloads to stringified IDs as well

This is equivalent to 591a9af356faf2d5c7e66e3ec715502796c875cd from
#5019, with an extra change for the addition to FeedManager#unpush.

* Ensure we have a status_id_seq sequence

Apparently this is not a given when specifying a custom ID function,
so now we ensure it gets created. This uses the generic version of this
function to more easily support adding additional tables with timestamp
IDs in the future, although it would be possible to cut this down to a
less generic version if necessary. It is only run during db:schema:load
or the relevant migration, so the overhead is extraordinarily minimal.

* Transition reblogs to new Redis format

This provides a one-way migration to transition old Redis reblog entries
into the new format, with a separate tracking entry for reblogs.

It is not invertible because doing so could (if timestamp IDs are used)
require a database query for each status in each users' feed, which is
likely to be a significant toll on major instances.

* Address review comments from @akihikodaki

No functional changes.

* Additional review changes

* Heredoc cleanup

* Run db:schema:load hooks for test in development

This matches the behavior in Rails'
ActiveRecord::Tasks::DatabaseTasks.each_current_configuration, which
would otherwise break `rake db:setup` in development.

It also moves some functionality out to a library, which will be a good
place to put additional related functionality in the near future.
2017-10-04 09:56:37 +02:00
Akihiko Odaki 63f0979799 Validate id of ActivityPub representations (#5114)
Additionally, ActivityPub::FetchRemoteStatusService no longer parses
activities.
OStatus::Activity::Creation no longer delegates to ActivityPub because
the provided ActivityPub representations are not signed while OStatus
representations are.
2017-10-04 01:13:48 +02:00
ThibG dfaa219f88 Fix HTTP responses for salmon and ActivityPub inbox processing (#5200)
* Return sensible HTTP status for ActivityPub inbox processing

* Return sensible HTTP status for salmon slap processing

* Return additional information to debug signature verification failures
2017-10-03 23:21:19 +02:00
Nolan Lawson 813c5f2f52 Add spec for emoji_index_light.js (#5199) 2017-10-03 20:54:38 +02:00
Nolan Lawson 334a446313 Fix emoji sequence bug in substring-trie (#5191)
Fixes #5188
2017-10-03 13:11:22 +02:00
Eugen Rochko 47ecd652d3 Make Chrome splash screen same color as web UI's background color (#5169) 2017-10-02 01:23:32 +02:00
Akihiko Odaki cdacac8c6c Fix order of paginated accounts in FollowerDomainsController and spec (#3357)
* Fix order of paginated accounts in FollowerDomainsController

Unordered pagination could result in unexpected behavior.

* Cover Settings::FollowerDomainsController more
2017-09-30 23:06:09 +02:00
Eugen Rochko eb605141ff Fix #5104 - GET /api/v1/apps/verify_credentials to confirm app works (#5112) 2017-09-30 22:05:42 +02:00
Eugen Rochko ebb8c89207 Upgrade to React 16 (#5119)
* Upgrade to React 16.0.0

* Disable some uncritical tests while chai-enzyme remains incompatible
2017-09-30 04:29:56 +02:00
Eugen Rochko f4ca116ea8 After 7 days of repeated delivery failures, give up on inbox (#5131)
- A successful delivery cancels it out
- An incoming delivery from account of the inbox cancels it out
2017-09-29 03:16:20 +02:00
Eugen Rochko 887cd94e96 Increase attachment descriptions to 420 characters (#5139)
Blaze it
2017-09-29 02:30:00 +02:00
Eugen Rochko 4ec1771165 Add ability to specify alternative text for media attachments (#5123)
* Fix #117 - Add ability to specify alternative text for media attachments

- POST /api/v1/media accepts `description` straight away
- PUT /api/v1/media/:id to update `description` (only for unattached ones)
- Serialized as `name` of Document object in ActivityPub
- Uploads form adjusted for better performance and description input

* Add tests

* Change undo button blend mode to difference
2017-09-28 15:31:31 +02:00
Ondřej Hruška 83bda6c1a8
Merge commit 'b9f59ebcc68e9da0a7158741a1a2ef3564e1321e' into merging-upstream 2017-09-28 09:18:35 +02:00
Ondřej Hruška 9330ea1f4d
Merge commit '4aea3f88a6d30f102a79c2da7fcfac96465ba1a8' into merging-upstream 2017-09-28 09:12:17 +02:00
Eugen Rochko e528114c53 Follow-up to #4582 and #5027, removing dead code (#5101) 2017-09-26 01:06:27 +02:00
Eugen Rochko cf7fbf2c56 Fix #5059 - Stop processing payload if it's from local account (#5100) 2017-09-26 01:06:13 +02:00
Eugen Rochko 510df0ac55 Fix emojify() by generating a mapping to existing Twemoji files (#5080)
A new rake task emojis:generate downloads a full list of valid
unicode sequences from unicode.org and checks it against existing
Twemoji files, finally generating a map from each sequence to the
existing file (e.g. when there's multiple ways an emoji can be
expressed). The map is dumped into app/javascript/mastodon/emoji_map.json

That file is loaded by emojione_light.js (now a misnomer) which
decorates it further with shortcodes taken from emoji-mart's index.
2017-09-25 18:36:33 +02:00
Akihiko Odaki 98936bfcdf Add missing validations in ActivityPub::Activity::Create (#5096) 2017-09-25 18:33:11 +02:00
Eugen Rochko 1e02ba111a Add emoji autosuggest (#5053)
* Add emoji autosuggest

Some credit goes to glitch-soc/mastodon#149

* Remove server-side shortcode->unicode conversion

* Insert shortcode when suggestion is custom emoji

* Remove remnant of server-side emojis

* Update style of autosuggestions

* Fix wrong emoji filenames generated in autosuggest item

* Do not lazy load emoji picker, as that no longer works

* Fix custom emoji autosuggest

* Fix multiple "Custom" categories getting added to emoji index, only add once
2017-09-23 14:47:32 +02:00
Eugen Rochko 293972f716 New API: GET /api/v1/custom_emojis to get a server's custom emojis (#5051) 2017-09-23 01:57:23 +02:00
Eugen Rochko 846cd4e838 Switch from EmojiOne to Twemoji, different emoji picker (#5046)
* Switch from EmojiOne to Twemoji, different emoji picker

* Make emoji-mart use a local spritesheet

* Fix emojify test

* yarn manage:translations
2017-09-23 01:41:00 +02:00
Akihiko Odaki 0de82dd316 Do not filter statuses with unknown languages (#5045) 2017-09-22 19:33:17 +02:00
Eugen Rochko 034fab39ab Make dropdowns render into portal, expand animation (#5018)
* Make dropdowns render into portal, expand animation

* Improve actions modal style
2017-09-22 04:59:17 +02:00
aschmitz 669fe9ee06 Change IDs to strings rather than numbers in API JSON output (#5019)
* Fix JavaScript interface with long IDs

Somewhat predictably, the JS interface handled IDs as numbers, which in
JS are IEEE double-precision floats. This loses some precision when
working with numbers as large as those generated by the new ID scheme,
so we instead handle them here as strings. This is relatively simple,
and doesn't appear to have caused any problems, but should definitely
be tested more thoroughly than the built-in tests. Several days of use
appear to support this working properly.

BREAKING CHANGE:

The major(!) change here is that IDs are now returned as strings by the
REST endpoints, rather than as integers. In practice, relatively few
changes were required to make the existing JS UI work with this change,
but it will likely hit API clients pretty hard: it's an entirely
different type to consume. (The one API client I tested, Tusky, handles
this with no problems, however.)

Twitter ran into this issue when introducing Snowflake IDs, and decided
to instead introduce an `id_str` field in JSON responses. I have opted
to *not* do that, and instead force all IDs to 64-bit integers
represented by strings in one go. (I believe Twitter exacerbated their
problem by rolling out the changes three times: once for statuses, once
for DMs, and once for user IDs, as well as by leaving an integer ID
value in JSON. As they said, "If you’re using the `id` field with JSON
in a Javascript-related language, there is a very high likelihood that
the integers will be silently munged by Javascript interpreters. In most
cases, this will result in behavior such as being unable to load or
delete a specific direct message, because the ID you're sending to the
API is different than the actual identifier associated with the
message." [1]) However, given that this is a significant change for API
users, alternatives or a transition time may be appropriate.

1: https://blog.twitter.com/developer/en_us/a/2011/direct-messages-going-snowflake-on-sep-30-2011.html

* Additional fixes for stringified IDs in JSON

These should be the last two. These were identified using eslint to try
to identify any plain casts to JavaScript numbers. (Some such casts are
legitimate, but these were not.)

Adding the following to .eslintrc.yml will identify casts to numbers:

~~~
  no-restricted-syntax:
  - warn
  - selector: UnaryExpression[operator='+'] > :not(Literal)
    message: Avoid the use of unary +
  - selector: CallExpression[callee.name='Number']
    message: Casting with Number() may coerce string IDs to numbers
~~~

The remaining three casts appear legitimate: two casts to array indices,
one in a server to turn an environment variable into a number.

* Back out RelationshipsController Change

This was made to make a test a bit less flakey, but has nothing to
do with this branch.

* Change internal streaming payloads to stringified IDs as well

Per
https://github.com/tootsuite/mastodon/pull/5019#issuecomment-330736452
we need these changes to send deleted status IDs as strings, not
integers.
2017-09-20 14:53:48 +02:00
Akihiko Odaki bb4d005a83 Introduce OStatus::TagManager (#5008) 2017-09-19 18:08:08 +02:00
Eugen Rochko 81cec35dbf Custom emoji (#4988)
* Custom emoji

- In OStatus: `<link rel="emoji" name="coolcat" href="http://..." />`
- In ActivityPub: `{ type: "Emoji", name: ":coolcat:", href: "http://..." }`
- In REST API: Status object includes `emojis` array (`shortcode`, `url`)
- Domain blocks with reject media stop emojis
- Emoji file up to 50KB
- Web UI handles custom emojis
- Static pages render custom emojis as `<img />` tags

Side effects:

- Undo #4500 optimization, as I needed to modify it to restore
  shortcode handling in emojify()
- Formatter#plaintext should now make sure stripped out line-breaks
  and paragraphs are replaced with newlines

* Fix emoji at the start not being converted
2017-09-19 02:42:40 +02:00
unarist dd6f9a1b82 Validate uri presence for remote status (#4985) 2017-09-17 15:21:57 +02:00
unarist 1eab53ee10 Fix an error when actor json couldn't be fetched in ResolveRemoteAccountService (#4979)
* Fix an error when actor json couldn't be fetched in ResolveRemoteAccountService

* Add specs
2017-09-17 11:54:23 +02:00
Akihiko Odaki 198a9a4fa4 Remove local_only scope in Status (#4977) 2017-09-17 05:29:43 +02:00
unarist ec36df97c4 Escape URL parts on formatting local status (#4975) 2017-09-16 21:33:52 +02:00
Eugen Rochko ad8e856a5b Fix error when following locked accounts (#4896)
(cherry picked from commit 0ef9d45d05)
2017-09-16 11:10:33 -05:00
Akihiko Odaki 48d77ea1eb Fix filterable_languages method of SettingsHelper (#4966) 2017-09-16 14:59:41 +02:00
Eugen Rochko 472df24579 When web UI URL used while logged out, redirect to static page (#4954) 2017-09-15 00:57:08 +02:00
ふぁぼ原 3816943e6b Enable to recognize most kinds of characters as URL paths (#4941) 2017-09-14 18:03:20 +02:00
Surinna Curtis d880b3182b minor code style fixes oops 2017-09-13 21:47:30 -05:00
Surinna Curtis 83b3a0389c Add more specs for the /api/v1/mutes/details endpoint 2017-09-13 21:47:30 -05:00
Surinna Curtis 70592cdaba Add a /api/v1/mutes/details route that just returns the array of mutes. 2017-09-13 21:47:30 -05:00
Surinna Curtis 500c465226 specs for MuteService notifications params 2017-09-13 21:47:30 -05:00
Surinna Curtis eaaf2170fe API support for muting notifications (and specs) 2017-09-13 21:47:30 -05:00
Surinna Curtis 5c2e1869f0 specs testing that hide_notifications in mutes actually hides notifications 2017-09-13 21:47:30 -05:00