* Fix streaming server using wrong property name for matching filters
Late in the PR, the `filter_results` property has been renamed to `filtered`,
but the change has not been reflected in the streaming server code.
* Fix filter_action attribute being an integer instead of a string
* Add model for custom filter keywords
* Use CustomFilterKeyword internally
Does not change the API
* Fix /filters/edit and /filters/new
* Add migration tests
* Remove whole_word column from custom_filters (covered by custom_filter_keywords)
* Redesign /filters
Instead of a list, present a card that displays more information and handles
multiple keywords per filter.
* Redesign /filters/new and /filters/edit to add and remove keywords
This adds a new gem dependency: cocoon, as well as a npm dependency:
cocoon-js-vanilla. Those are used to easily populate and remove form fields
from the user interface when manipulating multiple keyword filters at once.
* Add /api/v2/filters to edit filter with multiple keywords
Entities:
- `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context`
`keywords`
- `FilterKeyword`: `id`, `keyword`, `whole_word`
API endpoits:
- `GET /api/v2/filters` to list filters (including keywords)
- `POST /api/v2/filters` to create a new filter
`keywords_attributes` can also be passed to create keywords in one request
- `GET /api/v2/filters/:id` to read a particular filter
- `PUT /api/v2/filters/:id` to update a new filter
`keywords_attributes` can also be passed to edit, delete or add keywords in
one request
- `DELETE /api/v2/filters/:id` to delete a particular filter
- `GET /api/v2/filters/:id/keywords` to list keywords for a filter
- `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a
filter
- `GET /api/v2/filter_keywords/:id` to read a particular keyword
- `PUT /api/v2/filter_keywords/:id` to edit a particular keyword
- `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword
* Change from `irreversible` boolean to `action` enum
* Remove irrelevent `irreversible_must_be_within_context` check
* Fix /filters/new and /filters/edit with update for filter_action
* Fix Rubocop/Codeclimate complaining about task names
* Refactor FeedManager#phrase_filtered?
This moves regexp building and filter caching to the `CustomFilter` class.
This does not change the functional behavior yet, but this changes how the
cache is built, doing per-custom_filter regexps so that filters can be matched
independently, while still offering caching.
* Perform server-side filtering and output result in REST API
* Fix numerous filters_changed events being sent when editing multiple keywords at once
* Add some tests
* Use the new API in the WebUI
- use client-side logic for filters we have fetched rules for.
This is so that filter changes can be retroactively applied without
reloading the UI.
- use server-side logic for filters we haven't fetched rules for yet
(e.g. network error, or initial timeline loading)
* Minor optimizations and refactoring
* Perform server-side filtering on the streaming server
* Change the wording of filter action labels
* Fix issues pointed out by linter
* Change design of “Show anyway” link in accordence to review comments
* Drop “irreversible” filtering behavior
* Move /api/v2/filter_keywords to /api/v1/filters/keywords
* Rename `filter_results` attribute to `filtered`
* Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer
* Fix systemChannelId value in streaming server
* Simplify code by removing client-side filtering code
The simplifcation comes at a cost though: filters aren't retroactively
applied anymore.
Fixes#17541
- prefix JSON parsing error message by “Error parsing message from …”
- output user id if a user is logged in, IP address otherwise
- reduce log level from error to warning when a user is logged in, and to silly
otherwise
* Fix timeline streaming stopping for multiple sessions instead of one
Fixes#17256.
In updating the code for a newer version of node-redis, #17183 also broke
redis subscription management when multiple streaming clients subscribe to the
same channel.
This commit restores the redis subscription management code.
* Let node-redis actually handle the subscriptions
When checking for required OAuth scopes, an unexpected error could
happen due to missing (null-y) req.scopes. This commit fixes that by
checking if req.scopes are present before checking if any required
scopes are present, otherwise it skips that straight to rejection.
Eliminate need to have custom notifications filtering logic in the
streaming API code by publishing notifications into a separate stream
and then simply using the multi-stream capability to subscribe to that
stream when necessary
Fixes#14131
Our `mastodon:setup` task defaults to a blank password rather than the
absence of password, but some versions of Redis reject blank password
authentication when authentication is possible without a password.
The Ruby code only uses the Redis password when it's not blank, so
do the same for the node.js part.
* Send output on the server-sent events stream immediately so the client sees that it was successfully opened even if it doesn't have any messages.
Fix transparent SSE streaming for the public:local and hashtag:local stream types.
* Tell caches to never store server-sent events.
Previously, the server would attempt taking port/socket in worker
process, and if it was taken, fail, which made the master process
create a new worker. This led to really high CPU usage if the
streaming API was started when the port or socket were not
available.
Now, before clustering (forking) into worker processes, a test
server is created and then removed to check if it can be done.
* Add check for missing tag param in streaming API
Fixes error:
```
TypeError: Cannot read property 'toLowerCase' of undefined
at app.get (.../streaming/index.js:493:50)
```
* Fix code style issues
* Add conversations API
* Add web UI for conversations
* Add test for conversations API
* Add tests for ConversationAccount
* Improve web UI
* Rename ConversationAccount to AccountConversation
* Remove conversations on block and mute
* Change last_status_id to be a denormalization of status_ids
* Add optimistic locking
* Support UNIX domain socket for streaming service without using PORT
The use of UNIX domain socket for streaming service was not officially supported,
but it was made unofficial to use by setting a path to PORT.
From now on, SOCKET will be used just like setting for puma.
* Hundle relative path
* Added a timeline for Direct statuses
* Lists all Direct statuses you've sent and received
* Displayed in Getting Started
* Streaming server support for direct TL
* Changes to match other timelines in 2.0
On an unhandled worker exception, we should log the exception
and exit with nonzero status, instead of letting workers
silently fail and restarting them in an endless loop.
Note: we previously tried to handle the `'error'` signal.
That's not a signal Node fires; my patch traps `'uncaughtException'`,
which is what the code was _trying_ to do.
* make it possible to stream public timelines without authorization
* Fix
* Make eslint allow `value == null`
* Remove redundant line
* Improve style and revert .eslintrc.yml
* Fix streamWsEnd
* Show IP address instead of (anonymous user)
* Add missing semicolon
* Add structure for lists
* Add list timeline streaming API
* Add list APIs, bind list-account relation to follow relation
* Add API for adding/removing accounts from lists
* Add pagination to lists API
* Add pagination to list accounts API
* Adjust scopes for new APIs
- Creating and modifying lists merely requires "write" scope
- Fetching information about lists merely requires "read" scope
* Add test for wrong user context on list timeline
* Clean up tests
* Add redis key "subscribed:timeline:#{account.id}" to indicate active streaming API listeners exists.
* Add endpoint for notification only stream.
* Run PushUpdateWorker only for users uses Streaming API now.
* Move close hander streamTo(Http/Ws) -> stream(Http/Ws)End (Deal with #3370)
* Add stream type for stream start log message.
* Don't stream toots from users who have blocked the recipient user
This filter was already applied on the `/api/v1/timelines/public` API, but
not yet for the Streaming API.
* Boosted status' account_id doesn't need for filtering in streaming/index.js
This filtering is only for public/hashtag timelines, but boosts already filtered on FanOutOnWriteService because those timelines don't show boosts.
* Improve streaming API cluster logging
* Less verbose error middleware logging (stack trace useless there)
* Fix error logging
* Prevent potential issue
* Add missing "done()" in catch of Promise.all, websocket heartbeat re-implemented like in example
* I actually forgot a done(), the absolute madman
* Add <ostatus:conversation /> tag to Atom input/output
Only uses ref attribute (not href) because href would be
the alternate link that's always included also.
Creates new conversation for every non-reply status. Carries
over conversation for every reply. Keeps remote URIs verbatim,
generates local URIs on the fly like the rest of them.
* Conversation muting - prevents notifications that reference a conversation
(including replies, favourites, reblogs) from being created. API endpoints
/api/v1/statuses/:id/mute and /api/v1/statuses/:id/unmute
Currently no way to tell when a status/conversation is muted, so the web UI
only has a "disable notifications" button, doesn't work as a toggle
* Display "Dismiss notifications" on all statuses in notifications column, not just own
* Add "muted" as a boolean attribute on statuses JSON
For now always false on contained reblogs, since it's only relevant for
statuses returned from the notifications endpoint, which are not nested
Remove "Disable notifications" from detailed status view, since it's
only relevant in the notifications column
* Up max class length
* Remove pending test for conversation mute
* Add tests, clean up
* Rename to "mute conversation" and "unmute conversation"
* Raise validation error when trying to mute/unmute status without conversation
* Adding account domain blocks that filter notifications and public timelines
* Add tests for domain blocks in notifications, public timelines
Filter reblogs of blocked domains from home
* Add API for listing and creating account domain blocks
* API for creating/deleting domain blocks, tests for Status#ancestors
and Status#descendants, filter domain blocks from them
* Filter domains in streaming API
* Update account_domain_block_spec.rb
* Use babel-preset-env on Streaming Server
Change the streaming server to use babel-preset-env as well as asset compilation.
This shortens the load time at first boot.
* remove babel-plugin-lodash
* add redis-namespace gem
* namespace redis usage
* refactor redis namespace code to be less intrusive
previously : would be prepended to keys when the
REDIS_NAMESPACE env var was not set
now if it is not set the namespacing functions are
not used at all, which should prevent disruptions
when instances update.
* fix redis namespace variable style in streaming js
* remove trailing space
* final redis namespace style fix
* Allow puma to listen Unix socket
* streaming: Show the whole listening address instead of the port
Port is not always appropriate (e.g. Unix socket)
* streaming: Close server before exiting
This change especially allows to remove Unix socket before exiting.
* Make sure we start at least one streaming worker, even if there's only one CPU and we're in production mode.
* Oops, let's make sure we're counting cpus right too.
Emelia Smith