Commit Graph

136 Commits (1eca3d59682511156ab8a7a122e80f4cbb49990e)

Author SHA1 Message Date
Eugen Rochko 825dc3ca22 Add whitelist mode (#11291) 2019-07-30 11:10:46 +02:00
Eugen Rochko 31d4878fbb Change default interface of web and streaming from 0.0.0.0 to 127.0.0.1 (#11302) 2019-07-16 16:10:19 +02:00
Eugen Rochko fe299d3417 Change default interface of web and streaming from 0.0.0.0 to 127.0.0.1 (#11302) 2019-07-15 05:56:35 +02:00
Thibaut Girka 62eb42a9fa Fix streaming server crashing when updating filters 2019-06-18 16:50:04 +02:00
Thibaut Girka e8b45064b8 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- streaming/index.js
2019-05-24 15:23:52 +02:00
ThibG 7b1d582232 Improve streaming server security (#10818)
* Check OAuth token scopes in the streaming API

* Use Sec-WebSocket-Protocol instead of query string to pass WebSocket token

Inspired by https://github.com/kubevirt/kubevirt/issues/1242
2019-05-24 15:21:42 +02:00
Thibaut Girka d445d5833e Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/account.rb
- app/views/settings/profiles/show.html.haml
- spec/controllers/api/v1/accounts/credentials_controller_spec.rb

Conflicts were due to an increase in account bio length upstream, which
is already covered in glitch-soc through `MAX_BIO_CHARS`.
2019-05-23 19:01:30 +02:00
abcang 81768812f9 Migrate from uws to cws (#10805) 2019-05-22 18:19:16 +02:00
Thibaut Girka 33001891b6 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/services/post_status_service.rb
  Small conflict due to handling of instance-local toots.
  A subsequent change is required to ensure instance-local polls are not leaked
  through Update.
2019-03-11 11:23:50 +01:00
Eugen Rochko 5bc62cee47 Fix streaming API always attempting to use SSL with Postgres (#10231)
Fix #10223
2019-03-11 00:51:23 +01:00
Thibaut Girka 88d4b28714 Merge branch 'master' into glitch-soc/merge-upstream 2019-03-10 16:48:44 +01:00
Eugen Rochko d182fec01a Fix SSL configuration regression in streaming API (#10225)
* Fix SSL configuration regression in streaming API

Fix #10223

* Fix code style issues and integrate #10219

* Fix dumb thing
2019-03-10 16:00:54 +01:00
Thibaut Girka bed7758022 Merge branch 'master' into glitch-soc/merge-upstream 2019-03-08 20:15:57 +01:00
Sascha 291fbe21c2 config: add DB_SSLMODE for managed/remote PG (#10210)
* config: add DB_SSLMODE for managed/remote PG

* streaming: set PG sslmode, defaults to prefer
2019-03-08 14:36:28 +01:00
Thibaut Girka 10ec788f5d Only stream local-only toots to logged-in users 2018-11-13 18:55:57 +01:00
Gomasy af48120b62 Fix crash when using UNIX socket (#9036) 2018-10-21 16:41:33 +02:00
Eugen Rochko ac8afe9657 Check if port/socket is available before forking in Streaming API (#9023)
Previously, the server would attempt taking port/socket in worker
process, and if it was taken, fail, which made the master process
create a new worker. This led to really high CPU usage if the
streaming API was started when the port or socket were not
available.

Now, before clustering (forking) into worker processes, a test
server is created and then removed to check if it can be done.
2018-10-20 02:25:25 +02:00
Eugen Rochko d0bf87b689 Add check for missing tag param in streaming API (#8955)
* Add check for missing tag param in streaming API

Fixes error:

```
TypeError: Cannot read property 'toLowerCase' of undefined
at app.get (.../streaming/index.js:493:50)
```

* Fix code style issues
2018-10-11 19:24:43 +02:00
Eugen Rochko c9b5168ebd Add conversations API (#8832)
* Add conversations API

* Add web UI for conversations

* Add test for conversations API

* Add tests for ConversationAccount

* Improve web UI

* Rename ConversationAccount to AccountConversation

* Remove conversations on block and mute

* Change last_status_id to be a denormalization of status_ids

* Add optimistic locking
2018-10-07 23:44:58 +02:00
Eugen Rochko ec9e959ef9 Add health endpoint to streaming API (#8441)
GET /api/v1/streaming/health

Answers with OK. Fix #8337
2018-08-26 11:54:25 +02:00
MIYAGI Hikaru d164f9385a Support UNIX domain socket for streaming service without using PORT (#8217)
* Support UNIX domain socket for streaming service without using PORT

The use of UNIX domain socket for streaming service was not officially supported,
but it was made unofficial to use by setting a path to PORT.
From now on, SOCKET will be used just like setting for puma.

* Hundle relative path
2018-08-24 18:16:53 +02:00
Eugen Rochko 660f4d95d8 Fix streaming API still using filtered instead of chosen languages (#8009) 2018-07-14 03:59:31 +02:00
Yamagishi Kazutoshi ccc240b59c Add media timeline (#6631) 2018-05-21 12:43:38 +02:00
Kaito Sinclaire 447741e456 Direct messages column (#4514)
* Added a timeline for Direct statuses
* Lists all Direct statuses you've sent and received
* Displayed in Getting Started
* Streaming server support for direct TL

* Changes to match other timelines in 2.0
2018-04-18 13:09:06 +02:00
abcang ebc8475572 Perform processing that does not use the database before connecting to the database (#7168) 2018-04-17 13:49:09 +02:00
nullkal c40537c6ff Make sure call `done();` in the listener of public timeline for anonymous connection (#6009) 2017-12-13 14:27:36 +01:00
nullkal 44fb9f8030 Fix the condition in streaming listener (#6008) 2017-12-13 13:42:16 +01:00
erin d0f4496805 Improve error handling in streaming/index.js (#5968)
On an unhandled worker exception, we should log the exception
and exit with nonzero status, instead of letting workers
silently fail and restarting them in an endless loop.

Note: we previously tried to handle the `'error'` signal.
That's not a signal Node fires; my patch traps `'uncaughtException'`,
which is what the code was _trying_ to do.
2017-12-12 20:19:33 +01:00
nullkal 203d834c2e make it possible to stream public timelines without authorization (#5977)
* make it possible to stream public timelines without authorization

* Fix

* Make eslint allow `value == null`

* Remove redundant line

* Improve style and revert .eslintrc.yml

* Fix streamWsEnd

* Show IP address instead of (anonymous user)

* Add missing semicolon
2017-12-12 15:13:24 +01:00
Baptiste Gelez 51b2f7d79e Make it possible to bind streming service to 0.0.0.0 (#5744) 2017-11-18 04:44:19 +01:00
Eugen Rochko e149067561 Lists (#5703)
* Add structure for lists

* Add list timeline streaming API

* Add list APIs, bind list-account relation to follow relation

* Add API for adding/removing accounts from lists

* Add pagination to lists API

* Add pagination to list accounts API

* Adjust scopes for new APIs

- Creating and modifying lists merely requires "write" scope
- Fetching information about lists merely requires "read" scope

* Add test for wrong user context on list timeline

* Clean up tests
2017-11-18 00:16:48 +01:00
masarakki f76a224328 use-DB_NAME-in-development (#5430) 2017-10-17 11:45:37 +02:00
Eugen Rochko 6d12bcb225 Fix streaming API double-JSON-encoding status IDs for deletes (#5069) 2017-09-24 15:31:03 +02:00
voidSatisfaction 37dca6b7c7 Fix streaming url to lowercase (#4804) 2017-09-04 12:52:06 +02:00
Nolan Lawson c8ff11b785 Gracefully stop streaming server (#4103) 2017-07-07 20:01:00 +02:00
Eugen Rochko af19a1d90e Restore streaming API output format (#4100)
* Restore streaming API output format

Regression from #4090

* Remove whitespace
2017-07-07 16:56:52 +02:00
Yamagishi Kazutoshi 188098ce62 Stop using Babel with streaming server (#3950) 2017-06-26 04:49:39 +02:00
Takuya Yoshida 84d2df3ec6 Re-fix errorMiddleware (#3922) 2017-06-26 01:46:15 +02:00
unarist 104c7d7f96 Ignore DB_NAME for development env on streaming as well as rails side (#3948) 2017-06-26 01:45:50 +02:00
amazedkoumei a1fd5d7f61 more free pgconfig by .env (#3909)
* more free pgconfig for streaming by .env

* fix wrong default values

* database.yml read ENV as same as streaming server
2017-06-25 18:13:31 +02:00
Takuya Yoshida e81b5b25aa Fix errorMiddleware to prevent "TypeError: res.writeHead is not a function" (#3913)
* Fix errorMiddleware

* Add "eslint-disable-line no-unused-vars"
2017-06-23 19:22:02 +02:00
Yamagishi Kazutoshi 539dbaa049 Remove unused variables (#3906) 2017-06-23 16:05:04 +02:00
猫吸血鬼ディフリス / 猫ロキP 8f202bc639 Fix streaming server. Redis connection subscribe for each channel. (#3828) 2017-06-20 20:41:41 +02:00
Eugen Rochko 046307dfea Fix issue with some Node.js versions not having Array.prototype.includes (#3667)
by using Array.prototype.indexOf instead
2017-06-09 19:46:33 +02:00
Clworld 7e95d45c8d Execute PushUpdateWorker only for accounts who uses StreamingAPI just now. (#3278)
* Add redis key "subscribed:timeline:#{account.id}" to indicate active streaming API listeners exists.

* Add endpoint for notification only stream.

* Run PushUpdateWorker only for users uses Streaming API now.

* Move close hander streamTo(Http/Ws) -> stream(Http/Ws)End (Deal with #3370)

* Add stream type for stream start log message.
2017-06-03 20:50:53 +02:00
unarist 23e09cc6b7 [RFC] Return 401 for an authentication error on WebSockets (#3411)
* Return 401 for an authentication error on WebSocket

* Use upgradeReq instead of a custom object
2017-05-29 18:20:53 +02:00
unarist e5f9273631 Don't stream toots from users who have blocked the recipient user (#3300)
* Don't stream toots from users who have blocked the recipient user

This filter was already applied on the `/api/v1/timelines/public` API, but
not yet for the Streaming API.

* Boosted status' account_id doesn't need for filtering in streaming/index.js

This filtering is only for public/hashtag timelines, but boosts already filtered on FanOutOnWriteService because those timelines don't show boosts.
2017-05-29 18:01:08 +02:00
Clworld 0458982072 Re-add stream end log for WebSocket (#3397) 2017-05-28 19:14:44 +02:00
Eugen Rochko 1245d4fdbc Improve streaming API cluster logging (#3370)
* Improve streaming API cluster logging

* Less verbose error middleware logging (stack trace useless there)

* Fix error logging

* Prevent potential issue

* Add missing "done()" in catch of Promise.all, websocket heartbeat re-implemented like in example

* I actually forgot a done(), the absolute madman
2017-05-28 16:25:26 +02:00
Clworld 2f10e05e24 Reject revoked access_token on Streaming API. (#3367) 2017-05-27 23:27:54 +02:00
Eugen Rochko 8342091f47 Language filtering in streaming API (#3339) 2017-05-27 00:53:48 +02:00
Matt Jankowski 8dd89e9132 Use local default for postgres host in node streaming service (#3240)
This location varies across postgres installations, and it seems like the pg
package knows how to guess correctly on each system.
2017-05-22 20:27:24 -04:00
happycoloredbanana cb68926ebf Allow access token in URI (#3208) 2017-05-21 21:13:11 +02:00
Eugen Rochko d9797075e2 Adjust REDIS_URL usage in node_redis (#3183)
Resolves #2780
2017-05-20 21:06:09 +02:00
Yamagishi Kazutoshi e4ed0b155c Improve eslint rules (#3147)
* Add semi to ESLint rules

* Add padded-blocks to ESLint rules

* Add comma-dangle to ESLint rules

* add config/webpack and storyboard

* add streaming/

* yarn test:lint -- --fix
2017-05-20 17:31:47 +02:00
Eugen Rochko 0cafe62561 Account domain blocks (#2381)
* Add <ostatus:conversation /> tag to Atom input/output

Only uses ref attribute (not href) because href would be
the alternate link that's always included also.

Creates new conversation for every non-reply status. Carries
over conversation for every reply. Keeps remote URIs verbatim,
generates local URIs on the fly like the rest of them.

* Conversation muting - prevents notifications that reference a conversation
(including replies, favourites, reblogs) from being created. API endpoints
/api/v1/statuses/:id/mute and /api/v1/statuses/:id/unmute

Currently no way to tell when a status/conversation is muted, so the web UI
only has a "disable notifications" button, doesn't work as a toggle

* Display "Dismiss notifications" on all statuses in notifications column, not just own

* Add "muted" as a boolean attribute on statuses JSON

For now always false on contained reblogs, since it's only relevant for
statuses returned from the notifications endpoint, which are not nested

Remove "Disable notifications" from detailed status view, since it's
only relevant in the notifications column

* Up max class length

* Remove pending test for conversation mute

* Add tests, clean up

* Rename to "mute conversation" and "unmute conversation"

* Raise validation error when trying to mute/unmute status without conversation

* Adding account domain blocks that filter notifications and public timelines

* Add tests for domain blocks in notifications, public timelines
Filter reblogs of blocked domains from home

* Add API for listing and creating account domain blocks

* API for creating/deleting domain blocks, tests for Status#ancestors
and Status#descendants, filter domain blocks from them

* Filter domains in streaming API

* Update account_domain_block_spec.rb
2017-05-19 01:14:30 +02:00
Hayato IZUMI 93fcff82d8 Use REDIS_DB in streaming (#3094) 2017-05-17 15:36:34 +02:00
Ayumu AIZAWA ba8ea16858 [bug] wrong config setting for dababase. (#2984) 2017-05-11 11:23:24 +02:00
beatrix f88ff75a8d namespace redis usage (#2869)
* add redis-namespace gem

* namespace redis usage

* refactor redis namespace code to be less intrusive

previously : would be prepended to keys when the
REDIS_NAMESPACE env var was not set

now if it is not set the namespacing functions are
not used at all, which should prevent disruptions
when instances update.

* fix redis namespace variable style in streaming js

* remove trailing space

* final redis namespace style fix
2017-05-07 19:42:32 +02:00
Eshin Kunishima 0a9c2152c9 Replace ws with uws (#2807)
* Replace ws with uws

* fix indent

* Adjust indentation

* remove trailing space
2017-05-06 17:05:38 +02:00
abcang 2ceaa88ed0 fix DB_URL (#2778) 2017-05-04 15:53:44 +02:00
Eugen Rochko 0951a2f9f3 Clean up redis configuration. Allow using REDIS_URL to set advanced (#2732)
connection options instead of setting REDIS_HOST etc individually

Close #1986
2017-05-03 23:18:13 +02:00
Akihiko Odaki a28863bf95 Revert "Use PostgreSQL inheritance for blocks and mutes (#2520)" (#2634)
This reverts commit 7cd8e61b5a.
2017-04-30 08:49:24 -04:00
Akihiko Odaki 7cd8e61b5a Use PostgreSQL inheritance for blocks and mutes (#2520) 2017-04-30 00:27:31 +02:00
YOU 4b3a1e7790 Update index.js (#2612) 2017-04-29 14:35:16 +02:00
masarakki 4ca919cb50 refactor-cores (#2308) 2017-04-22 15:11:56 +02:00
Akihiko Odaki 4e097cc6c0 Allow to listen Unix socket (#2085)
* Allow puma to listen Unix socket

* streaming: Show the whole listening address instead of the port

Port is not always appropriate (e.g. Unix socket)

* streaming: Close server before exiting

This change especially allows to remove Unix socket before exiting.
2017-04-21 19:24:31 +02:00
178inaba 00a666a205 Fix a bug where the hashtag could not be specified with the streaming API (#2258) 2017-04-21 16:45:14 +02:00
kuro5hin a80568a305 Fix streaming server worker count (#2082)
* Make sure we start at least one streaming worker, even if there's only one CPU and we're in production mode.

* Oops, let's make sure we're counting cpus right too.
2017-04-18 15:58:22 +02:00
geta6 4146b86beb Improve streaming server with cluster (#1970) 2017-04-17 04:32:30 +02:00
Gavin Mogan a710eec95c Dev Tooling fixes (eslint/editorconfig) (#1398)
* Add eslint to dev dependancies so it gets installed for the repo

yarn add --dev eslint babel-eslint eslint-plugin-reac

project specific version of eslint, you can globally install eslint-cli
if you want the global runtime, or add .bin to your path

* fix eslint errors about inconsitent returns

* eslint ignore the same as git ignore. allows for eslint .

* Add editorconfig file so everyones editor will be setup to follow the same standards
2017-04-11 00:36:03 +02:00
Eugen Rochko 6966854f3d Add heartbeat to websockets streaming API connections 2017-04-02 21:27:14 +02:00
Eugen Rochko 9989890e2e Fix #231 - Muting 2017-03-02 18:49:32 +01:00
Eugen Rochko 1f106f7955 Make streaming API use one pattern-matching redis pubsub connection
Refresh timelines when streaming API reconnects in the UI
2017-02-07 14:39:08 +01:00
Eugen Rochko eb9934c88c Add streaming API channels for local-only statuses 2017-02-06 23:46:14 +01:00
Eugen Rochko b8ab1bdd70 Add unique request IDs to streaming API to improve logs 2017-02-05 23:37:25 +01:00
Eugen Rochko 728817fba2 Add tracking of delay to streaming API 2017-02-05 03:19:04 +01:00
Eugen Rochko 9bd2b6be86 Make the streaming API also handle websockets (because trying to get the browser EventSource interface to
work flawlessly was a nightmare). WARNING: This commit makes the web UI connect to the streaming API instead
of ActionCable like before. This means that if you are upgrading, you should set that up beforehand.
2017-02-04 00:34:31 +01:00
Eugen Rochko c6f0dfadf7 Add CORS to the streaming API 2017-02-03 18:27:42 +01:00
Eugen Rochko c51f8dd4e4 Fix streaming API queries 2017-02-02 17:10:59 +01:00
Eugen Rochko b860cbcbdf Adding documentation for streaming API. Fixing filtering bug 2017-02-02 16:35:06 +01:00
Eugen Rochko 2db5a5a981 Make node read the right .env file 2017-02-02 16:11:36 +01:00
Eugen Rochko d10d56275b Add redis configuration to streaming API 2017-02-02 15:29:41 +01:00
Eugen Rochko 8f7aac3076 Fix potential resource leaks in streaming server 2017-02-02 15:20:31 +01:00
Eugen Rochko 999ecc9d69 Add logging and filtering to the node.js streaming API 2017-02-02 13:56:14 +01:00
Eugen Rochko d4601c094f Add streaming API in Node.js 2017-02-02 01:31:09 +01:00