Commit Graph

95 Commits (349e244fb80b721d101c8469ed8a575172e76ab0)

Author SHA1 Message Date
Claire 7ea520b56c Fix timeline streaming stopping for multiple sessions instead of one (#17259)
* Fix timeline streaming stopping for multiple sessions instead of one

Fixes #17256.

In updating the code for a newer version of node-redis, #17183 also broke
redis subscription management when multiple streaming clients subscribe to the
same channel.

This commit restores the redis subscription management code.

* Let node-redis actually handle the subscriptions
2022-01-07 19:50:12 +01:00
Lerk 9003270cb5 Finish update of node-redis (#17183)
* fix streaming redis client

* use console.error instead of console.log

* follow node-redis migration guide

https://github.com/redis/node-redis/blob/master/docs/v3-to-v4.md

* fix config options for node-redis

* keep indentation

* Update streaming/index.js

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2021-12-25 22:55:06 +01:00
Sasha Sorokin 9906bc20d2 fix(streaming): req.scopes can be nullable (#16823)
When checking for required OAuth scopes, an unexpected error could
happen due to missing (null-y) req.scopes. This commit fixes that by
checking if req.scopes are present before checking if any required
scopes are present, otherwise it skips that straight to rejection.
2021-10-13 05:02:55 +02:00
Eugen Rochko 4752c0a8d3 Refactor notifications to go through a separate stream in streaming API (#16765)
Eliminate need to have custom notifications filtering logic in the
streaming API code by publishing notifications into a separate stream
and then simply using the multi-stream capability to subscribe to that
stream when necessary
2021-09-26 13:23:28 +02:00
Eugen Rochko 759cca2d31 Fix "cb is not a function" error in streaming API server (#16134)
Third argument of `ping` is the callback

Regression from #15932
2021-05-02 14:30:26 +02:00
Eugen Rochko f22c178647 Change log level of worker start/end to warn in streaming API (#16110) 2021-05-01 23:19:18 +02:00
Claire 90cd93f49a Switch from deprecated ClusterWS/cws to ws package (#15932)
* Switch from deprecated ClusterWS/cws to ws package

Fixes #15184

Co-authored-by: Edho Arief <me@nanaya.pro>

* Make bufferutil and utf-8-validate optional dependencies

Co-authored-by: Edho Arief <me@nanaya.pro>
2021-03-24 09:37:41 +01:00
abcang 144e606ec9 Fixed ESLint error (#15214)
* eslint --fix

* fix consistent-return

* fix promise/catch-or-return

* ignore import rule
2020-11-23 17:35:14 +01:00
Eugen Rochko 7e2920d9b2 Fix streaming API allowing connections to persist after access token invalidation (#15111)
Fix #14816
2020-11-12 23:05:24 +01:00
fuyu 86d6f45786 Fix crash streaming process when receive invalid json (#14859) 2020-09-22 15:30:41 +02:00
Eugen Rochko 0043da909b Fix destructuring error when unsubscribing without subscribing (#14566) 2020-08-12 15:36:07 +02:00
Eugen Rochko 3d32dacbcb Add support for managing multiple stream subscriptions in a single connection (#14524) 2020-08-11 18:24:59 +02:00
ThibG e303673059 Change streaming server to treat blank redis password as password-less auth (#14135)
Fixes #14131

Our `mastodon:setup` task defaults to a blank password rather than the
absence of password, but some versions of Redis reject blank password
authentication when authentication is possible without a password.

The Ruby code only uses the Redis password when it's not blank, so
do the same for the node.js part.
2020-06-24 22:25:23 +02:00
Eugen Rochko b864e78db8 Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
Takeshi Umeda 861661100c Add remote only to public timeline (#13504)
* Add remote only to public timeline

* Fix code style
2020-05-10 10:36:18 +02:00
Ben Lubar 6598ab989b minor server-sent events fixes (#12945)
* Send output on the server-sent events stream immediately so the client sees that it was successfully opened even if it doesn't have any messages.

Fix transparent SSE streaming for the public:local and hashtag:local stream types.

* Tell caches to never store server-sent events.
2020-01-24 20:51:33 +01:00
Eugen Rochko 825dc3ca22 Add whitelist mode (#11291) 2019-07-30 11:10:46 +02:00
Eugen Rochko fe299d3417 Change default interface of web and streaming from 0.0.0.0 to 127.0.0.1 (#11302) 2019-07-15 05:56:35 +02:00
ThibG 7b1d582232 Improve streaming server security (#10818)
* Check OAuth token scopes in the streaming API

* Use Sec-WebSocket-Protocol instead of query string to pass WebSocket token

Inspired by https://github.com/kubevirt/kubevirt/issues/1242
2019-05-24 15:21:42 +02:00
abcang 81768812f9 Migrate from uws to cws (#10805) 2019-05-22 18:19:16 +02:00
Eugen Rochko 5bc62cee47 Fix streaming API always attempting to use SSL with Postgres (#10231)
Fix #10223
2019-03-11 00:51:23 +01:00
Eugen Rochko d182fec01a Fix SSL configuration regression in streaming API (#10225)
* Fix SSL configuration regression in streaming API

Fix #10223

* Fix code style issues and integrate #10219

* Fix dumb thing
2019-03-10 16:00:54 +01:00
Sascha 291fbe21c2 config: add DB_SSLMODE for managed/remote PG (#10210)
* config: add DB_SSLMODE for managed/remote PG

* streaming: set PG sslmode, defaults to prefer
2019-03-08 14:36:28 +01:00
Gomasy af48120b62 Fix crash when using UNIX socket (#9036) 2018-10-21 16:41:33 +02:00
Eugen Rochko ac8afe9657 Check if port/socket is available before forking in Streaming API (#9023)
Previously, the server would attempt taking port/socket in worker
process, and if it was taken, fail, which made the master process
create a new worker. This led to really high CPU usage if the
streaming API was started when the port or socket were not
available.

Now, before clustering (forking) into worker processes, a test
server is created and then removed to check if it can be done.
2018-10-20 02:25:25 +02:00
Eugen Rochko d0bf87b689 Add check for missing tag param in streaming API (#8955)
* Add check for missing tag param in streaming API

Fixes error:

```
TypeError: Cannot read property 'toLowerCase' of undefined
at app.get (.../streaming/index.js:493:50)
```

* Fix code style issues
2018-10-11 19:24:43 +02:00
Eugen Rochko c9b5168ebd Add conversations API (#8832)
* Add conversations API

* Add web UI for conversations

* Add test for conversations API

* Add tests for ConversationAccount

* Improve web UI

* Rename ConversationAccount to AccountConversation

* Remove conversations on block and mute

* Change last_status_id to be a denormalization of status_ids

* Add optimistic locking
2018-10-07 23:44:58 +02:00
Eugen Rochko ec9e959ef9 Add health endpoint to streaming API (#8441)
GET /api/v1/streaming/health

Answers with OK. Fix #8337
2018-08-26 11:54:25 +02:00
MIYAGI Hikaru d164f9385a Support UNIX domain socket for streaming service without using PORT (#8217)
* Support UNIX domain socket for streaming service without using PORT

The use of UNIX domain socket for streaming service was not officially supported,
but it was made unofficial to use by setting a path to PORT.
From now on, SOCKET will be used just like setting for puma.

* Hundle relative path
2018-08-24 18:16:53 +02:00
Eugen Rochko 660f4d95d8 Fix streaming API still using filtered instead of chosen languages (#8009) 2018-07-14 03:59:31 +02:00
Yamagishi Kazutoshi ccc240b59c Add media timeline (#6631) 2018-05-21 12:43:38 +02:00
Kaito Sinclaire 447741e456 Direct messages column (#4514)
* Added a timeline for Direct statuses
* Lists all Direct statuses you've sent and received
* Displayed in Getting Started
* Streaming server support for direct TL

* Changes to match other timelines in 2.0
2018-04-18 13:09:06 +02:00
abcang ebc8475572 Perform processing that does not use the database before connecting to the database (#7168) 2018-04-17 13:49:09 +02:00
nullkal c40537c6ff Make sure call `done();` in the listener of public timeline for anonymous connection (#6009) 2017-12-13 14:27:36 +01:00
nullkal 44fb9f8030 Fix the condition in streaming listener (#6008) 2017-12-13 13:42:16 +01:00
erin d0f4496805 Improve error handling in streaming/index.js (#5968)
On an unhandled worker exception, we should log the exception
and exit with nonzero status, instead of letting workers
silently fail and restarting them in an endless loop.

Note: we previously tried to handle the `'error'` signal.
That's not a signal Node fires; my patch traps `'uncaughtException'`,
which is what the code was _trying_ to do.
2017-12-12 20:19:33 +01:00
nullkal 203d834c2e make it possible to stream public timelines without authorization (#5977)
* make it possible to stream public timelines without authorization

* Fix

* Make eslint allow `value == null`

* Remove redundant line

* Improve style and revert .eslintrc.yml

* Fix streamWsEnd

* Show IP address instead of (anonymous user)

* Add missing semicolon
2017-12-12 15:13:24 +01:00
Baptiste Gelez 51b2f7d79e Make it possible to bind streming service to 0.0.0.0 (#5744) 2017-11-18 04:44:19 +01:00
Eugen Rochko e149067561 Lists (#5703)
* Add structure for lists

* Add list timeline streaming API

* Add list APIs, bind list-account relation to follow relation

* Add API for adding/removing accounts from lists

* Add pagination to lists API

* Add pagination to list accounts API

* Adjust scopes for new APIs

- Creating and modifying lists merely requires "write" scope
- Fetching information about lists merely requires "read" scope

* Add test for wrong user context on list timeline

* Clean up tests
2017-11-18 00:16:48 +01:00
masarakki f76a224328 use-DB_NAME-in-development (#5430) 2017-10-17 11:45:37 +02:00
Eugen Rochko 6d12bcb225 Fix streaming API double-JSON-encoding status IDs for deletes (#5069) 2017-09-24 15:31:03 +02:00
voidSatisfaction 37dca6b7c7 Fix streaming url to lowercase (#4804) 2017-09-04 12:52:06 +02:00
Nolan Lawson c8ff11b785 Gracefully stop streaming server (#4103) 2017-07-07 20:01:00 +02:00
Eugen Rochko af19a1d90e Restore streaming API output format (#4100)
* Restore streaming API output format

Regression from #4090

* Remove whitespace
2017-07-07 16:56:52 +02:00
Yamagishi Kazutoshi 188098ce62 Stop using Babel with streaming server (#3950) 2017-06-26 04:49:39 +02:00
Takuya Yoshida 84d2df3ec6 Re-fix errorMiddleware (#3922) 2017-06-26 01:46:15 +02:00
unarist 104c7d7f96 Ignore DB_NAME for development env on streaming as well as rails side (#3948) 2017-06-26 01:45:50 +02:00
amazedkoumei a1fd5d7f61 more free pgconfig by .env (#3909)
* more free pgconfig for streaming by .env

* fix wrong default values

* database.yml read ENV as same as streaming server
2017-06-25 18:13:31 +02:00
Takuya Yoshida e81b5b25aa Fix errorMiddleware to prevent "TypeError: res.writeHead is not a function" (#3913)
* Fix errorMiddleware

* Add "eslint-disable-line no-unused-vars"
2017-06-23 19:22:02 +02:00
Yamagishi Kazutoshi 539dbaa049 Remove unused variables (#3906) 2017-06-23 16:05:04 +02:00