Commit Graph

287 Commits (main)

Author SHA1 Message Date
Emelia Smith 4949b6da58
Fix streaming eslint configuration (#28055) 2023-11-28 14:09:21 +00:00
Michael Stanclift a80530d1df
Dockerfile rewrite based on Ruby image with performance optimizations and size reduction, dedicated Streaming image (#26850)
Co-authored-by: “Michael <“mx@vmstan.com>
Co-authored-by: Emelia Smith <ThisIsMissEm@users.noreply.github.com>
2023-11-28 10:04:40 +00:00
renovate[bot] 73ab080608
Update dependency jsdom to v23 (#28075)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2023-11-27 10:06:40 +00:00
renovate[bot] 1ee3314f58
Update dependency @types/npmlog to v7 (#28048)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2023-11-23 10:28:52 +00:00
renovate[bot] 998f068499
Update Yarn to v4.0.2 (#27857)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2023-11-14 17:52:34 +00:00
Emelia Smith 15b2d7eec5
Split streaming server from web server (#24702) 2023-11-14 17:43:20 +00:00
Claire 4a9cdfb928 Merge commit '774e1189d26fffd914107a4236f6287043c988f8' into glitch-soc/merge-upstream 2023-10-09 21:12:28 +02:00
Renaud Chaput 774e1189d2
Change `eslint` config to autofix missing comma and indentation in JS files (#26711) 2023-10-09 13:38:29 +02:00
Claire e9a181c52c Merge commit 'e95d25e1013b6328457b81bd98e8d6a841d45ec2' into glitch-soc/merge-upstream 2023-10-08 16:53:05 +02:00
Emelia Smith f68d540271
Fix websocket connections being incorrectly decremented twice on errors (#27238) 2023-10-02 13:21:43 +02:00
Claire 22ee4e057d Merge commit 'a83615edc9fc1ce3363ddcf1fc676806421a5f65' into glitch-soc/merge-upstream
Conflicts:
- `streaming/index.js`:
  Upstream has added a parameter to `streamFrom`, while glitch-soc had an
  extra parameter.
  Ported upstream changes.
2023-09-19 12:59:23 +02:00
Emelia Smith fbfceb9c77
Add additional metrics for streaming (#26945) 2023-09-19 12:25:30 +02:00
Claire 4ed629cd7e Merge commit 'be991f1d18006a4820c1e9ca6625bf2bd2bfedac' into glitch-soc/merge-upstream
Conflicts:
- `app/models/form/admin_settings.rb`:
  Upstream added the notion of overriden settings, while we had extra code for
  pseudo-settings (only used to combine flavour and skin in a single select
  field).
  Ported upstream changes.
- `config/i18n-tasks.yml`:
  Upstream added `simple_form.overridden` to `ignore_unused`,
  we had `simple_form.glitch_only`.
  Added `simple_form.glitch_only` as well.
2023-09-02 14:44:47 +02:00
Gabriel Simmer be991f1d18
Move to ioredis for streaming (#26581)
Co-authored-by: Emelia Smith <ThisIsMissEm@users.noreply.github.com>
2023-09-01 17:44:28 +02:00
Claire 1b5d3fdc5e Merge commit 'ee702e36e58d638bcf75b2eae2ca86499693465e' into glitch-soc/merge-upstream 2023-08-24 20:55:28 +02:00
Emelia Smith 6375e390af
Fix: support both DATABASE_URL and DB_PASS (#26295) 2023-08-18 15:05:35 +02:00
Claire 6530f9709c Merge commit '121443c0fca383268b8022c048dd137994785aff' into glitch-soc/main
Conflicts:
- `.rubocop_todo.yml`:
  Upstream regenerated this file, glitch-soc had a specific ignore.
2023-08-13 18:47:15 +02:00
Emelia Smith 65096c1ccd
Change streaming `/metrics` to include additional metrics (#26299) 2023-08-04 16:11:30 +02:00
Claire 92fa9d34b0 Merge commit '3554c527954441fd924586a49c7d99a89101ac7e' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/authorize_interactions_controller.rb`:
  Small conflict due to our theming system.
- `streaming/index.js`:
  Upstream refactored part of the streaming server.
  We had some extra logic for handling local-only posts.
  Applied the refactor.
2023-07-30 16:11:55 +02:00
Claire 454d8567d4 Merge commit '1814990a3d117555153321216fa593e4d9e84de3' into glitch-soc/merge-upstream 2023-07-30 15:54:34 +02:00
Renaud Chaput 3554c52795
Fix missing return values in streaming (#26233) 2023-07-28 19:11:58 +02:00
Claire 446821a0dc
Swap debug statements in streaming server (#26231) 2023-07-28 17:59:31 +02:00
Emelia Smith 81cdc0f972
Fix: Streaming server memory leak in HTTP EventSource cleanup (#26228) 2023-07-28 12:06:29 +02:00
Emelia Smith ddaf200c78
Refactor streaming's filtering logic & improve documentation (#26213) 2023-07-27 15:38:18 +02:00
Claire 1814990a3d
Fix wrong filters sometimes applying in streaming (#26159) 2023-07-27 15:12:10 +02:00
Claire 65cbcce997 Merge commit '39110d1d0af5e3d9cf452ae47496a52797249fd0' into glitch-soc/merge-upstream 2023-06-18 10:36:14 +02:00
Claire 42a0898f16 Merge commit '3a2a15c6ea4d4603469861ed9be09da12a122e45' into glitch-soc/merge-upstream
Conflicts:
- `app/views/settings/preferences/appearance/show.html.haml`:
  Conflict because glitch-soc does not have a theme selector here.
2023-06-10 18:42:29 +02:00
Emelia Smith 215081240f
Add logging of websocket send errors (#25280) 2023-06-10 18:35:57 +02:00
Emelia Smith c81f59583c
Fix logging of messages that are binary before closing their connection (#25361) 2023-06-10 18:24:37 +02:00
Claire 8c26d49cbd Merge commit '16dd3f08c1e5396d5f9ff3f13417901bc4e4b8b9' into glitch-soc/merge-upstream
Conflicts:
- `app/views/settings/preferences/appearance/show.html.haml`:
  Upstream fixed a translation bug in the theme selector that is absent from
  glitch-soc due to our different theming system.
  Discarded upstream changes.
- `streaming/index.js`:
  Upstream changed the signature of a function to change its return type.
  This is not a real conflict, the conflict being caused by an extra
  argument in glitch-soc's code.
  Applied upstream's change while keeping our extra argument.
2023-06-10 17:18:36 +02:00
Emelia Smith 16dd3f08c1
Fix performance of streaming by parsing message JSON once (#25278) 2023-06-09 19:29:16 +02:00
Claire 1cd7f5ff17 Merge commit 'd27216dc4616d80659c0cc5d2a55394e0e1ae874' into glitch-soc/merge-upstream
Conflicts:
- `.eslintrc.js`:
  Upstream moved a configuration block in which we had added a glitch-only
  path.
  Moved the configuration block as upstream did.
- other files:
  Upstream reordered imports, and those files had different ones.
  Kept our version and reordered imports using the same rules.
2023-05-28 15:54:31 +02:00
Renaud Chaput d27216dc46
Enforce import order with ESLint (#25096) 2023-05-23 17:15:17 +02:00
Claire 2d029dedd9 Merge commit '0ad2413b35287958f59073a5b63aecc659a64d98' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/styles/mastodon/forms.scss`:
  Conflict because we ran eslint autofix on upstream files.
- `config/initializers/content_security_policy.rb`:
  Code style changes but we have a different version.
  Kept our version.
- `streaming/index.js`:
  Upstream fixed a typo close to glitch-soc-only code.
  Applied upstream's changes.
2023-05-08 15:28:36 +02:00
Claire 3fb7fe14c6
Fix some of the Javascript linting issues, as well as bugs and unneeded divergences with upstream (#2208)
* Run eslint --fix

* Fix linting issues in video player and reduce divergences with upstream

This includes a behavior change of not auto-looping videos anymore. I don't
remember loops being ever intended, and they have been removed from upstream
a while ago, but we somehow missed the change.

* Fix lint issues in `app/javascript/flavours/glitch/selectors/index.js`

Those were basically caused by dead code that isn't present upstream, so
that brings us closer to upstream as well.

* Fix linting issue and bug in streaming/index.js

* Fix linting issues in config/webpack/shared.js

* Fix unused import in flavours/glitch/features/ui/index.js

* Fix linting issues and reduce divergences from upstream in flavours/glitch/features/ui/components/video_modal.jsx

* Fix linting issues in flavours/glitch/reducers

* Fix linting issues in glitch-soc onboarding modal

* Fix linting issues in flavours/glitch/features/ui/components/navigation_panel.jsx

* Remove dead code for unused local setting navbar_under

* Fix various linting issues

* Fix linting issues in flavours/glitch/components/scrollable_list.jsx and reduce divergences with upstream
2023-05-07 18:22:25 +02:00
Nick Schonning f22b4e556c
Add eslint-plugin-jsdoc (#24719) 2023-04-30 02:29:54 +02:00
Claire 12b935fadf Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `.github/dependabot.yml`:
  Updated upstream, removed in glitch-soc to disable noise.
  Kept removed.
- `CODE_OF_CONDUCT.md`:
  Upstream updated to a new version of the covenant, but I have not read it
  yet, so kept unchanged.
- `Gemfile.lock`:
  Not a real conflict, one upstream dependency updated textually too close to
  the glitch-soc only `hcaptcha` dependency.
  Applied upstream changes.
- `app/controllers/admin/base_controller.rb`:
  Minor conflict due to glitch-soc's theming system.
  Applied upstream changes.
- `app/controllers/application_controller.rb`:
  Minor conflict due to glitch-soc's theming system.
  Applied upstream changes.
- `app/controllers/disputes/base_controller.rb`:
  Minor conflict due to glitch-soc's theming system.
  Applied upstream changes.
- `app/controllers/relationships_controller.rb`:
  Minor conflict due to glitch-soc's theming system.
  Applied upstream changes.
- `app/controllers/statuses_cleanup_controller.rb`:
  Minor conflict due to glitch-soc's theming system.
  Applied upstream changes.
- `app/helpers/application_helper.rb`:
  Minor conflict due to glitch-soc's theming system.
  Applied upstream changes.
- `app/javascript/mastodon/features/compose/components/compose_form.jsx`:
  Upstream added a highlight animation for onboarding, while we changed the
  max character limit.
  Applied our local changes on top of upstream's new version.
- `app/views/layouts/application.html.haml`:
  Minor conflict due to glitch-soc's theming system.
  Applied upstream changes.
- `stylelint.config.js`:
  Upstream added ignore paths, glitch-soc had extra ignore paths.
  Added the same paths as upstream.
2023-04-29 10:44:56 +02:00
Emelia Smith 301e03eb8c
Remove clustering from streaming API (#24655) 2023-04-26 11:37:51 +02:00
Claire 3ef5f62abf Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `.github/workflows/build-image.yml`:
  Upstream switched to pushing to both DockerHub and GitHub Container
  Repository, while glitch-soc was already pushing to the latter only.
  Updated our configuration to be slightly more consistent with upstream's
  naming and styling, but kept our behavior.
- `Gemfile.lock`:
  Updated dependencies textually too close to glitch-soc only hcaptcha
  dependency.
  Updated dependencies as upstream did.
- `README.md`:
  Upstream updated its README, but we have a completely different one.
  Kept our README, though it probably should be reworked at some point.
- `app/views/auth/sessions/two_factor.html.haml`:
  Minor style fix upstream that's on a line glitch-soc removed because
  of its different theming system.
  Kept our file as is.
- `spec/controllers/health_controller_spec.rb`:
  This file apparently did not exist upstream, upstream created it with
  different contents but it is functionally the same.
  Switched to upstream's version of the file.
- `spec/presenters/instance_presenter_spec.rb`:
  Upstream changed the specs around `GITHUB_REPOSITORY`, while glitch-soc
  had its own code because it's a fork and does not have the same default
  source URL.
  Took upstream's change, but with glitch-soc's repo as the default case.
- `yarn.lock`:
  Upstream dependencies textually too close to a glitch-soc only one.
  Updated dependencies as upstream did.
2023-03-15 09:16:10 +01:00
Claire 174352716a
Disable anonymous access to the streaming API (#23989) 2023-03-06 21:00:10 +01:00
Claire 7623e18124 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Upstream README has been changed, but we have a completely different one.
  Kept our `README.md`.
- `lib/sanitize_ext/sanitize_config.rb`:
  Upstream added support for more incoming HTML tags (a large subset of what
  glitch-soc accepts).
  Change the code style to match upstream's but otherwise do not change our
  code.
- `spec/lib/sanitize_config_spec.rb`:
  Upstream added support for more incoming HTML tags (a large subset of what
  glitch-soc accepts).
  Kept our version, since the tests are mostly glitch-soc's, except for cases
  which are purposefuly different.
2023-03-05 20:46:56 +01:00
Eugen Rochko be1792e1d4
Fix streaming API not being usable without `DATABASE_URL` (#23960) 2023-03-05 01:52:12 +01:00
Ramūns Usovs 0872f3e3d7
Allow streaming to connect to postgress with self-signed certs (#21431) 2023-03-03 21:01:18 +01:00
Claire ce84d163cc Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `.prettierignore`:
  Upstream added a line at the end, glitch-soc had extra entries at the end.
  Added upstream's new line before glitch-soc's.
- `Gemfile.lock`:
  Upstream updated dependencies while glitch-soc has an extra one (hcaptcha).
  Updated dependencies like upstream did.
- `app/controllers/api/v1/statuses_controller.rb`:
  Not a real conflict, upstream added a parameter (`allowed_mentions`) where
  glitch-soc already had an extra one (`content_type`).
  Added upstream's new parameter.
- `app/javascript/styles/fonts/roboto-mono.scss`:
  A lot of lines were changed upstream due to code style changes, and a lot
  of those lines had path changes to accomodate glitch-soc's theming system.
  Applied upstream's style changes.
- `app/javascript/styles/fonts/roboto.scss`:
  A lot of lines were changed upstream due to code style changes, and a lot
  of those lines had path changes to accomodate glitch-soc's theming system.
  Applied upstream's style changes.
2023-02-13 19:35:35 +01:00
emilweth 7c45145430
fix metrics format (#23520) 2023-02-11 03:00:44 +01:00
emilweth c1b823368c
dot is not allowed (#23519) 2023-02-11 02:15:07 +01:00
Claire 85558a5e18 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Minor upstream change, our README is completely different.
  Kept ours.
- `lib/tasks/assets.rake`:
  glitch-soc has extra code to deal with its theming system,
  upstream changed a line that exists in glitch-soc.
  Applied upstream changes.
2023-02-09 12:46:12 +01:00
Eugen Rochko c5a4d8c82d
Add information about database pool to metrics endpoint in streaming API (#23469) 2023-02-09 11:24:16 +01:00
Eugen Rochko 4cb7647516
Add database pool configuration over `DB_POOL` to streaming API (#23470)
Add connection timeout of 15s
2023-02-09 11:20:59 +01:00
Eugen Rochko 01584f03e8
Add metrics endpoint to streaming API (#23388) 2023-02-03 21:29:23 +01:00
Claire b248aa4d3e Merge branch 'main' into glitch-soc/merge-upstream 2022-12-21 15:59:39 +01:00
Nick Schonning 06b68490d1
Enable eslint:recommended ruleset (#22433)
* Enable ESLint recommended ruleset

* Disable failing ESLint recommended rules

* Remove rules shadowed by eslint:recommended
2022-12-19 00:51:37 +09:00
Claire 66461ae79a Merge branch 'main' into glitch-soc/merge-upstream 2022-11-15 02:10:37 +01:00
Claire 8c3c6dcae9
Fix filters from other users being used in the streaming service (#20719) 2022-11-15 02:09:58 +01:00
Claire 32c70d2f09 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/views/admin/announcements/edit.html.haml`:
  Upstream change too close to theming-related glitch-soc change.
  Ported upstream changes.
- `app/views/admin/announcements/new.html.haml`
  Upstream change too close to theming-related glitch-soc change.
  Ported upstream changes.
2022-11-14 08:35:55 +01:00
Claire cd5e98dbdb
Fix public/local timeline posts not being properly filtered (#20567)
* Fix streaming server using wrong property name for matching filters

Late in the PR, the `filter_results` property has been renamed to `filtered`,
but the change has not been reflected in the streaming server code.

* Fix filter_action attribute being an integer instead of a string
2022-11-13 20:59:49 +01:00
Claire cd87d7dcef Merge branch 'main' into glitch-soc/merge-upstream
- `.env.production.sample`:
  Our sample config file is very different from upstream since it is much more
  complete. Upstream added documentation for a few env variables.
  Copied the new variables/documentation from upstream.
- `app/lib/feed_manager.rb`:
  Upstream added a timeline type (hashtags), while glitch-soc already had an
  extra one (direct messages). Not really a conflict but textually close
  changes.
  Ported upstream's changes.
- `app/models/custom_emoji.rb`:
  Upstream upped the custom emoji size limit, while glitch-soc had configurable
  limits.
  Upped the default limits accordingly.
- `streaming/index.js`:
  Upstream reworked how hastags were normalized. Minor conflict due to
  glitch-soc's handling of instance-local posts.
  Ported upstream's changes.
2022-07-17 22:07:20 +02:00
Eugen Rochko e7aa2be828
Change how hashtags are normalized (#18795)
* Change how hashtags are normalized

* Fix tests
2022-07-13 15:03:28 +02:00
Claire fe5f6bc7ed Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `.github/workflows/build-image.yml`:
  Fix erroneous deletion in a previous merge.
- `Gemfile`:
  Conflict caused by glitch-soc-only hCaptcha dependency
- `app/controllers/auth/sessions_controller.rb`:
  Minor conflict due to glitch-soc's theming system.
- `app/controllers/filters_controller.rb`:
  Minor conflict due to glitch-soc's theming system.
- `app/serializers/rest/status_serializer.rb`:
  Minor conflict due to glitch-soc having an extra `local_only` property
2022-06-28 11:11:18 +02:00
Claire 02851848e9
Revamp post filtering system (#18058)
* Add model for custom filter keywords

* Use CustomFilterKeyword internally

Does not change the API

* Fix /filters/edit and /filters/new

* Add migration tests

* Remove whole_word column from custom_filters (covered by custom_filter_keywords)

* Redesign /filters

Instead of a list, present a card that displays more information and handles
multiple keywords per filter.

* Redesign /filters/new and /filters/edit to add and remove keywords

This adds a new gem dependency: cocoon, as well as a npm dependency:
cocoon-js-vanilla. Those are used to easily populate and remove form fields
from the user interface when manipulating multiple keyword filters at once.

* Add /api/v2/filters to edit filter with multiple keywords

Entities:
- `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context`
  `keywords`
- `FilterKeyword`: `id`, `keyword`, `whole_word`

API endpoits:
- `GET /api/v2/filters` to list filters (including keywords)
- `POST /api/v2/filters` to create a new filter
  `keywords_attributes` can also be passed to create keywords in one request
- `GET /api/v2/filters/:id` to read a particular filter
- `PUT /api/v2/filters/:id` to update a new filter
  `keywords_attributes` can also be passed to edit, delete or add keywords in
   one request
- `DELETE /api/v2/filters/:id` to delete a particular filter
- `GET /api/v2/filters/:id/keywords` to list keywords for a filter
- `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a
   filter
- `GET /api/v2/filter_keywords/:id` to read a particular keyword
- `PUT /api/v2/filter_keywords/:id` to edit a particular keyword
- `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword

* Change from `irreversible` boolean to `action` enum

* Remove irrelevent `irreversible_must_be_within_context` check

* Fix /filters/new and /filters/edit with update for filter_action

* Fix Rubocop/Codeclimate complaining about task names

* Refactor FeedManager#phrase_filtered?

This moves regexp building and filter caching to the `CustomFilter` class.

This does not change the functional behavior yet, but this changes how the
cache is built, doing per-custom_filter regexps so that filters can be matched
independently, while still offering caching.

* Perform server-side filtering and output result in REST API

* Fix numerous filters_changed events being sent when editing multiple keywords at once

* Add some tests

* Use the new API in the WebUI

- use client-side logic for filters we have fetched rules for.
  This is so that filter changes can be retroactively applied without
  reloading the UI.
- use server-side logic for filters we haven't fetched rules for yet
  (e.g. network error, or initial timeline loading)

* Minor optimizations and refactoring

* Perform server-side filtering on the streaming server

* Change the wording of filter action labels

* Fix issues pointed out by linter

* Change design of “Show anyway” link in accordence to review comments

* Drop “irreversible” filtering behavior

* Move /api/v2/filter_keywords to /api/v1/filters/keywords

* Rename `filter_results` attribute to `filtered`

* Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer

* Fix systemChannelId value in streaming server

* Simplify code by removing client-side filtering code

The simplifcation comes at a cost though: filters aren't retroactively
applied anymore.
2022-06-28 09:42:13 +02:00
Claire 002d2729fb Merge branch 'main' into glitch-soc/merge-upstream 2022-04-25 17:09:32 +02:00
Yamagishi Kazutoshi 5781d1db84
Fix parsing `TRUSTED_PROXY_IP` (#18051) 2022-04-19 08:11:58 +01:00
Claire 59864ff495 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `.github/dependabot.yml`:
  Upstream modified it, but we deleted it in glitch-soc.
  Keep it deleted.
2022-03-22 17:00:38 +01:00
Claire f29458da1d
Fix streaming server sometimes silently dropping subscriptions (#17841) 2022-03-21 19:08:29 +01:00
Claire f224237862 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `db/schema.rb`:
  Conflict due to glitch-soc adding the `content_type` column on status edits
  and thus having a different schema version number.
  Solved by taking upstream's schema version number, as it is higher than
  glitch-soc's.
2022-02-17 10:58:44 +01:00
Claire 6ea80ba2a2
Change streaming server error messages when failing to parse client input (#17559)
Fixes #17541

- prefix JSON parsing error message by “Error parsing message from …”
- output user id if a user is logged in, IP address otherwise
- reduce log level from error to warning when a user is logged in, and to silly
  otherwise
2022-02-16 14:37:26 +01:00
Claire fca50e4b50 Merge branch 'main' into glitch-soc/merge-upstream 2022-01-07 19:51:47 +01:00
Claire bc066d8936
Fix timeline streaming stopping for multiple sessions instead of one (#17259)
* Fix timeline streaming stopping for multiple sessions instead of one

Fixes #17256.

In updating the code for a newer version of node-redis, #17183 also broke
redis subscription management when multiple streaming clients subscribe to the
same channel.

This commit restores the redis subscription management code.

* Let node-redis actually handle the subscriptions
2022-01-07 19:50:12 +01:00
Claire 09c61e1376 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  We have completely different contents. Kept our version.
- `package.json`:
  Not a real conflict, just an upstream dependency udpated
  textually too close to a glitch-soc-only dependency.
  Updated dependencies like upstream.
- `streaming/index.js`:
  Conflict due to code style changes on parts that were
  modified in glitch-soc to handle local-only toots.
  Changed style according to upstream.
2021-12-27 15:16:18 +01:00
Lerk 4d1eaf3e6e
Finish update of node-redis (#17183)
* fix streaming redis client

* use console.error instead of console.log

* follow node-redis migration guide

https://github.com/redis/node-redis/blob/master/docs/v3-to-v4.md

* fix config options for node-redis

* keep indentation

* Update streaming/index.js

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2021-12-25 22:55:06 +01:00
Claire 1665b84285 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `package.json`:
  Not really a conflict, just a glitch-soc-only dependency textually too close
  to an updated upstream one.
- `yarn.lock`:
  Not really a conflict, just a glitch-soc-only dependency textually too close
  to an updated upstream one.
2021-10-13 15:50:24 +02:00
Sasha Sorokin 6c88ebfd4b
fix(streaming): req.scopes can be nullable (#16823)
When checking for required OAuth scopes, an unexpected error could
happen due to missing (null-y) req.scopes. This commit fixes that by
checking if req.scopes are present before checking if any required
scopes are present, otherwise it skips that straight to rejection.
2021-10-13 05:02:55 +02:00
Claire 3622110778 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `streaming/index.js`:
  Filtering code for streaming notifications has been refactored upstream, but
  glitch-soc had similar code for local-only toots in the same places.
  Ported upstream changes, but did not refactor local-only filtering.
2021-09-26 18:28:59 +02:00
Eugen Rochko a0d4129893
Refactor notifications to go through a separate stream in streaming API (#16765)
Eliminate need to have custom notifications filtering logic in the
streaming API code by publishing notifications into a separate stream
and then simply using the multi-stream capability to subscribe to that
stream when necessary
2021-09-26 13:23:28 +02:00
Claire 50b430d9a2 Merge branch 'main' into glitch-soc/merge-upstream
- `app/views/statuses/_simple_status.html.haml`:
  Small markup change in glitch-soc, on a line that has been modified by
  upstream. Ported upstream changes.
2021-05-07 18:21:59 +02:00
Eugen Rochko c5c46dd6ee
Fix "cb is not a function" error in streaming API server (#16134)
Third argument of `ping` is the callback

Regression from #15932
2021-05-02 14:30:26 +02:00
Eugen Rochko aafe65a142
Change log level of worker start/end to warn in streaming API (#16110) 2021-05-01 23:19:18 +02:00
Claire f60c99a8fb Merge branch 'main' into glitch-soc/merge-upstream 2021-03-24 10:53:50 +01:00
Claire 49814d5799
Switch from deprecated ClusterWS/cws to ws package (#15932)
* Switch from deprecated ClusterWS/cws to ws package

Fixes #15184

Co-authored-by: Edho Arief <me@nanaya.pro>

* Make bufferutil and utf-8-validate optional dependencies

Co-authored-by: Edho Arief <me@nanaya.pro>
2021-03-24 09:37:41 +01:00
Thibaut Girka e71b16ae2b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  glitch-soc-only dependency textually too close to updated upstream
  dependencies. Updated to upsteam dependencies.
2020-11-25 15:31:45 +01:00
abcang a2da02626e
Fixed ESLint error (#15214)
* eslint --fix

* fix consistent-return

* fix promise/catch-or-return

* ignore import rule
2020-11-23 17:35:14 +01:00
Thibaut Girka de5cc20dd8 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/concerns/sign_in_token_authentication_concern.rb`:
  Conflict caused because of glitch-soc's theming system.
  Took upstream's new code and applied the theming system changes on top
  of it.
- `app/controllers/concerns/two_factor_authentication_concern.rb`:
  Conflict caused because of glitch-soc's theming system.
  Took upstream's new code and applied the theming system changes on top
  of it.
2020-11-14 00:30:36 +01:00
Eugen Rochko aa10200e58
Fix streaming API allowing connections to persist after access token invalidation (#15111)
Fix #14816
2020-11-12 23:05:24 +01:00
Thibaut Girka a7aedebc31 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, upstream updated dependencies that were too close to
  glitch-soc-only ones in the file.
- `app/controllers/oauth/authorized_applications_controller.rb`:
  Upstream changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/controllers/settings/base_controller.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/controllers/settings/sessions_controller.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/models/user.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc not preventing moved accounts from logging
  in.
  Ported upstream changes while keeping the ability for moved accounts to log
  in.
- `app/policies/status_policy.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's local-only toots.
  Ported upstream changes.
- `app/serializers/rest/account_serializer.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's ability  to hide followers count.
  Ported upstream changes.
- `app/services/process_mentions_service.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's local-only toots.
  Ported upstream changes.
- `package.json`:
  Not a real conflict, upstream updated dependencies that were too close to
  glitch-soc-only ones in the file.
2020-09-28 14:13:30 +02:00
fuyu e39d97f700
Fix crash streaming process when receive invalid json (#14859) 2020-09-22 15:30:41 +02:00
Thibaut Girka da62e350e0 Merge branch 'master' into glitch-soc/merge-upstream 2020-08-13 22:17:29 +02:00
Eugen Rochko 01647b8acb
Fix destructuring error when unsubscribing without subscribing (#14566) 2020-08-12 15:36:07 +02:00
Thibaut Girka 78fa15d08f Merge branch 'master' into glitch-soc/master
Conflicts:
- `streaming/index.js`:
  Upstream entirely refactored it.
  Ported our changes to upstream's refactor. Hopefuly.
2020-08-11 19:19:27 +02:00
Eugen Rochko ef057584fd
Add support for managing multiple stream subscriptions in a single connection (#14524) 2020-08-11 18:24:59 +02:00
Thibaut Girka c41da2e962 Fix streaming server only working for streams allowing local-only toots 2020-07-07 20:31:45 +02:00
Thibaut Girka d973e11061 Add UI option to show local-only toots in public timeline 2020-07-06 21:12:06 +02:00
Thibaut Girka a01674fb2b Merge branch 'master' into glitch-soc/merge-upstream 2020-06-26 13:02:14 +02:00
ThibG ddcdddd6fc
Change streaming server to treat blank redis password as password-less auth (#14135)
Fixes #14131

Our `mastodon:setup` task defaults to a blank password rather than the
absence of password, but some versions of Redis reject blank password
authentication when authentication is possible without a password.

The Ruby code only uses the Redis password when it's not blank, so
do the same for the node.js part.
2020-06-24 22:25:23 +02:00
Thibaut Girka 12c8ac9e14 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/activitypub/collections_controller.rb`:
  Conflict due to glitch-soc having to take care of local-only
  pinned toots in that controller.
  Took upstream's changes and restored the local-only special
  handling.
- `app/controllers/auth/sessions_controller.rb`:
  Minor conflicts due to the theming system, applied upstream
  changes, adapted the following two files for glitch-soc's
  theming system:
  - `app/controllers/concerns/sign_in_token_authentication_concern.rb`
  - `app/controllers/concerns/two_factor_authentication_concern.rb`
- `app/services/backup_service.rb`:
  Minor conflict due to glitch-soc having to handle local-only
  toots specially. Applied upstream changes and restored
  the local-only special handling.
- `app/views/admin/custom_emojis/index.html.haml`:
  Minor conflict due to the theming system.
- `package.json`:
  Upstream dependency updated, too close to a glitch-soc-only
  dependency in the file.
- `yarn.lock`:
  Upstream dependency updated, too close to a glitch-soc-only
  dependency in the file.
2020-06-09 10:39:20 +02:00
Eugen Rochko 5d8398c8b8
Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
Thibaut Girka 4a70792b4a Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, just a glitch-soc-only dependency too close to a
  dependency that got updated upstream. Updated as well.
- `app/models/status.rb`:
  Not a real conflict, just a change too close to glitch-soc-changed code
  for optionally showing boosts in public timelines.
  Applied upstream changes.
- `app/views/layouts/application.html.haml`:
  Upstream a new, static CSS file, conflict due to glitch-soc's theming
  system, include the file regardless of the theme.
- `config/initializers/content_security_policy.rb`:
  Upstream dropped 'unsafe-inline' from the 'style-src' directive, but
  both files are very different. Removed 'unsafe-inline' as well.
2020-05-10 16:19:56 +02:00
Takeshi Umeda 26b08a3c54
Add remote only to public timeline (#13504)
* Add remote only to public timeline

* Fix code style
2020-05-10 10:36:18 +02:00
Thibaut Girka 61f699c0e9 Merge branch 'master' into glitch-soc/merge-upstream 2020-01-25 10:04:41 +01:00
Ben Lubar 0dfba0884e minor server-sent events fixes (#12945)
* Send output on the server-sent events stream immediately so the client sees that it was successfully opened even if it doesn't have any messages.

Fix transparent SSE streaming for the public:local and hashtag:local stream types.

* Tell caches to never store server-sent events.
2020-01-24 20:51:33 +01:00
Thibaut Girka 24968d20a0 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/directories_controller.rb
- package.json
- yarn.lock
2019-07-30 12:22:33 +02:00
Eugen Rochko 24552b5160
Add whitelist mode (#11291) 2019-07-30 11:10:46 +02:00
Eugen Rochko be053560e9 Change default interface of web and streaming from 0.0.0.0 to 127.0.0.1 (#11302) 2019-07-16 16:10:19 +02:00
Eugen Rochko e7353c47db
Change default interface of web and streaming from 0.0.0.0 to 127.0.0.1 (#11302) 2019-07-15 05:56:35 +02:00
Thibaut Girka 48ec6abaca Fix streaming server crashing when updating filters 2019-06-18 16:50:04 +02:00
Thibaut Girka 652147a3f4 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- streaming/index.js
2019-05-24 15:23:52 +02:00
ThibG d63c3c0cef Improve streaming server security (#10818)
* Check OAuth token scopes in the streaming API

* Use Sec-WebSocket-Protocol instead of query string to pass WebSocket token

Inspired by https://github.com/kubevirt/kubevirt/issues/1242
2019-05-24 15:21:42 +02:00
Thibaut Girka c0dc247bce Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/account.rb
- app/views/settings/profiles/show.html.haml
- spec/controllers/api/v1/accounts/credentials_controller_spec.rb

Conflicts were due to an increase in account bio length upstream, which
is already covered in glitch-soc through `MAX_BIO_CHARS`.
2019-05-23 19:01:30 +02:00
abcang ca6c93a2f5 Migrate from uws to cws (#10805) 2019-05-22 18:19:16 +02:00
Thibaut Girka 89bee860cd Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/services/post_status_service.rb
  Small conflict due to handling of instance-local toots.
  A subsequent change is required to ensure instance-local polls are not leaked
  through Update.
2019-03-11 11:23:50 +01:00
Eugen Rochko 13a7f05030
Fix streaming API always attempting to use SSL with Postgres (#10231)
Fix #10223
2019-03-11 00:51:23 +01:00
Thibaut Girka dfe7322922 Merge branch 'master' into glitch-soc/merge-upstream 2019-03-10 16:48:44 +01:00
Eugen Rochko dbeab5a036
Fix SSL configuration regression in streaming API (#10225)
* Fix SSL configuration regression in streaming API

Fix #10223

* Fix code style issues and integrate #10219

* Fix dumb thing
2019-03-10 16:00:54 +01:00
Thibaut Girka 674fe32f97 Merge branch 'master' into glitch-soc/merge-upstream 2019-03-08 20:15:57 +01:00
Sascha 42e733681a config: add DB_SSLMODE for managed/remote PG (#10210)
* config: add DB_SSLMODE for managed/remote PG

* streaming: set PG sslmode, defaults to prefer
2019-03-08 14:36:28 +01:00
Thibaut Girka a100f05687 Only stream local-only toots to logged-in users 2018-11-13 18:55:57 +01:00
Gomasy 8d70d3de38 Fix crash when using UNIX socket (#9036) 2018-10-21 16:41:33 +02:00
Eugen Rochko 369cc5f555
Check if port/socket is available before forking in Streaming API (#9023)
Previously, the server would attempt taking port/socket in worker
process, and if it was taken, fail, which made the master process
create a new worker. This led to really high CPU usage if the
streaming API was started when the port or socket were not
available.

Now, before clustering (forking) into worker processes, a test
server is created and then removed to check if it can be done.
2018-10-20 02:25:25 +02:00
Eugen Rochko 18e7ef6eda
Add check for missing tag param in streaming API (#8955)
* Add check for missing tag param in streaming API

Fixes error:

```
TypeError: Cannot read property 'toLowerCase' of undefined
at app.get (.../streaming/index.js:493:50)
```

* Fix code style issues
2018-10-11 19:24:43 +02:00
Eugen Rochko 774ac47373
Add conversations API (#8832)
* Add conversations API

* Add web UI for conversations

* Add test for conversations API

* Add tests for ConversationAccount

* Improve web UI

* Rename ConversationAccount to AccountConversation

* Remove conversations on block and mute

* Change last_status_id to be a denormalization of status_ids

* Add optimistic locking
2018-10-07 23:44:58 +02:00
Eugen Rochko f37fafe30b
Add health endpoint to streaming API (#8441)
GET /api/v1/streaming/health

Answers with OK. Fix #8337
2018-08-26 11:54:25 +02:00
MIYAGI Hikaru a083604d1d Support UNIX domain socket for streaming service without using PORT (#8217)
* Support UNIX domain socket for streaming service without using PORT

The use of UNIX domain socket for streaming service was not officially supported,
but it was made unofficial to use by setting a path to PORT.
From now on, SOCKET will be used just like setting for puma.

* Hundle relative path
2018-08-24 18:16:53 +02:00
Eugen Rochko 0180037dfb
Fix streaming API still using filtered instead of chosen languages (#8009) 2018-07-14 03:59:31 +02:00
Yamagishi Kazutoshi 7403e5d306 Add media timeline (#6631) 2018-05-21 12:43:38 +02:00
Kaito Sinclaire 156b916caf Direct messages column (#4514)
* Added a timeline for Direct statuses
* Lists all Direct statuses you've sent and received
* Displayed in Getting Started
* Streaming server support for direct TL

* Changes to match other timelines in 2.0
2018-04-18 13:09:06 +02:00
abcang 609bf93029 Perform processing that does not use the database before connecting to the database (#7168) 2018-04-17 13:49:09 +02:00
nullkal ccf4f170de Make sure call `done();` in the listener of public timeline for anonymous connection (#6009) 2017-12-13 14:27:36 +01:00
nullkal 90e7da16a0 Fix the condition in streaming listener (#6008) 2017-12-13 13:42:16 +01:00
erin c986218c3a Improve error handling in streaming/index.js (#5968)
On an unhandled worker exception, we should log the exception
and exit with nonzero status, instead of letting workers
silently fail and restarting them in an endless loop.

Note: we previously tried to handle the `'error'` signal.
That's not a signal Node fires; my patch traps `'uncaughtException'`,
which is what the code was _trying_ to do.
2017-12-12 20:19:33 +01:00
nullkal cfea28216f make it possible to stream public timelines without authorization (#5977)
* make it possible to stream public timelines without authorization

* Fix

* Make eslint allow `value == null`

* Remove redundant line

* Improve style and revert .eslintrc.yml

* Fix streamWsEnd

* Show IP address instead of (anonymous user)

* Add missing semicolon
2017-12-12 15:13:24 +01:00
Baptiste Gelez 2864e5e077 Make it possible to bind streming service to 0.0.0.0 (#5744) 2017-11-18 04:44:19 +01:00
Eugen Rochko 24cafd73a2
Lists (#5703)
* Add structure for lists

* Add list timeline streaming API

* Add list APIs, bind list-account relation to follow relation

* Add API for adding/removing accounts from lists

* Add pagination to lists API

* Add pagination to list accounts API

* Adjust scopes for new APIs

- Creating and modifying lists merely requires "write" scope
- Fetching information about lists merely requires "read" scope

* Add test for wrong user context on list timeline

* Clean up tests
2017-11-18 00:16:48 +01:00
masarakki 0e0c6b1b4b use-DB_NAME-in-development (#5430) 2017-10-17 11:45:37 +02:00
Eugen Rochko 909a6d4661 Fix streaming API double-JSON-encoding status IDs for deletes (#5069) 2017-09-24 15:31:03 +02:00
voidSatisfaction b6f3869f8d Fix streaming url to lowercase (#4804) 2017-09-04 12:52:06 +02:00
Nolan Lawson 3f82d8b979 Gracefully stop streaming server (#4103) 2017-07-07 20:01:00 +02:00
Eugen Rochko ebd2dde688 Restore streaming API output format (#4100)
* Restore streaming API output format

Regression from #4090

* Remove whitespace
2017-07-07 16:56:52 +02:00
Yamagishi Kazutoshi 285038972b Stop using Babel with streaming server (#3950) 2017-06-26 04:49:39 +02:00
Takuya Yoshida e5563843a2 Re-fix errorMiddleware (#3922) 2017-06-26 01:46:15 +02:00
unarist c972e1ee1f Ignore DB_NAME for development env on streaming as well as rails side (#3948) 2017-06-26 01:45:50 +02:00
amazedkoumei 87efa38721 more free pgconfig by .env (#3909)
* more free pgconfig for streaming by .env

* fix wrong default values

* database.yml read ENV as same as streaming server
2017-06-25 18:13:31 +02:00
Takuya Yoshida 0c44316b22 Fix errorMiddleware to prevent "TypeError: res.writeHead is not a function" (#3913)
* Fix errorMiddleware

* Add "eslint-disable-line no-unused-vars"
2017-06-23 19:22:02 +02:00
Yamagishi Kazutoshi eff9416469 Remove unused variables (#3906) 2017-06-23 16:05:04 +02:00
猫吸血鬼ディフリス / 猫ロキP d8ec832806 Fix streaming server. Redis connection subscribe for each channel. (#3828) 2017-06-20 20:41:41 +02:00
Eugen Rochko 0464602978 Fix issue with some Node.js versions not having Array.prototype.includes (#3667)
by using Array.prototype.indexOf instead
2017-06-09 19:46:33 +02:00
Clworld dab8fc4584 Execute PushUpdateWorker only for accounts who uses StreamingAPI just now. (#3278)
* Add redis key "subscribed:timeline:#{account.id}" to indicate active streaming API listeners exists.

* Add endpoint for notification only stream.

* Run PushUpdateWorker only for users uses Streaming API now.

* Move close hander streamTo(Http/Ws) -> stream(Http/Ws)End (Deal with #3370)

* Add stream type for stream start log message.
2017-06-03 20:50:53 +02:00
unarist 9a81be0d37 [RFC] Return 401 for an authentication error on WebSockets (#3411)
* Return 401 for an authentication error on WebSocket

* Use upgradeReq instead of a custom object
2017-05-29 18:20:53 +02:00
unarist 91c789ec63 Don't stream toots from users who have blocked the recipient user (#3300)
* Don't stream toots from users who have blocked the recipient user

This filter was already applied on the `/api/v1/timelines/public` API, but
not yet for the Streaming API.

* Boosted status' account_id doesn't need for filtering in streaming/index.js

This filtering is only for public/hashtag timelines, but boosts already filtered on FanOutOnWriteService because those timelines don't show boosts.
2017-05-29 18:01:08 +02:00
Clworld 45837c533e Re-add stream end log for WebSocket (#3397) 2017-05-28 19:14:44 +02:00
Eugen Rochko 425d02287a Improve streaming API cluster logging (#3370)
* Improve streaming API cluster logging

* Less verbose error middleware logging (stack trace useless there)

* Fix error logging

* Prevent potential issue

* Add missing "done()" in catch of Promise.all, websocket heartbeat re-implemented like in example

* I actually forgot a done(), the absolute madman
2017-05-28 16:25:26 +02:00
Clworld 2e429c0c25 Reject revoked access_token on Streaming API. (#3367) 2017-05-27 23:27:54 +02:00