Commit Graph

658 Commits (f145662c053b7a043039bc14fdfff92d7f7e6125)

Author SHA1 Message Date
ThibG a3c9062153 Add border around 🐞 emoji (#14712) 2020-08-31 23:30:27 +02:00
Takeshi Umeda e0b8201f67 Fix an error when file_file_size is nil in tootctl media remove (#14657) 2020-08-24 18:09:46 +02:00
ThibG 8baad22f68 Improve email address validation (#14565)
* Increase DNS timeout from 1 second to 5 seconds for MX check

1 seconds is rather short when using a recursive DNS resolver which
hasn't got a cached result already available. Use 5 seconds instead,
which is the timeout value we use for outgoing HTTP queries.

* Add more precise error messages for invalid e-mail addresses
2020-08-12 12:40:25 +02:00
ThibG ccb975df83 Fix `tootctl media` commands not handling snowflake ids for media_attachments (#14536) 2020-08-10 01:51:06 +02:00
ThibG 20d6fb7f1b Fix thumbnail color extraction (#14464)
* Fix contrast calculation for thumbnail color extraction

Luminance calculation was using 0-255 RGB values instead of 0-1 sRGB values,
leading to incorrectly-computed contrast values.

Since we use ColorDiff already, just use its XYZ colorspace conversion code
to get the value.

* Require at least 3:1 contrast for both accent and foreground colors

* Lower required contrast for the accent color
2020-08-02 18:47:44 +02:00
ThibG 0dd641dcea Change content-type to be always computed from file data (#14452)
* Change content-type to be always computed from file data

Restore previous behavior, detecting the content-type isn't very
expensive, and some instances may serve files as application/octet-stream
regardless of their true type, making fetching media from them fail, while
it used to work pre-3.2.0.

* Add test
2020-08-02 11:21:10 +02:00
Eugen Rochko f089dd55ef Bump version to 3.2.0 (#14395) 2020-07-27 00:07:26 +02:00
Eugen Rochko 1959db3fd1 Bump version to 3.2.0rc2 (#14382) 2020-07-24 18:47:25 +02:00
Takeshi Umeda 9e16f0f1f6 Fix mimetype returning nil (#14356) 2020-07-19 13:09:47 +02:00
Eugen Rochko 6391e1e302 Bump version to 3.2.0rc1 (#14312) 2020-07-15 21:07:27 +02:00
Eugen Rochko 09caf7e22e Change `tootctl search deploy` algorithm (#14300) 2020-07-14 18:10:35 +02:00
Eugen Rochko c3ef166b61 Bump version to 3.1.5 (#14284) 2020-07-10 21:07:30 +02:00
ThibG 5adec9f3b6 Fix ogg vorbis files with a cover art not being correctly processed (#14255) 2020-07-07 16:32:14 +02:00
Eugen Rochko 8517a5fdb4 Add color extraction for audio thumbnails (#14209) 2020-07-05 18:28:25 +02:00
Eugen Rochko ae28f94a7a Fix audio uploads without embedded image (#14203) 2020-07-03 03:05:32 +02:00
Eugen Rochko ca238c879d Change Redis#exists calls to Redis#exists? to avoid deprecation warning (#14191) 2020-07-01 19:05:21 +02:00
Eugen Rochko 36e5fd547b Fix remote files not using Content-Type header, streaming (#14184) 2020-06-30 23:58:02 +02:00
Eugen Rochko a3ce01a102 Add customizable thumbnails for audio and video attachments (#14145)
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
2020-06-29 13:56:55 +02:00
Eugen Rochko 42d2f45db1 Remove the terms blacklist and whitelist from UX (#14149)
Localization strings:

- "Whitelist mode" -> "Limited federation mode"
- "Blacklist e-mail domain" -> "Block e-mail domain"
- "Whitelist domain" -> "Allow domain for federation"

...And so on

Environment variables (backwards-compatible):

- `WHITELIST_MODE` -> `LIMITED_FEDERATION_MODE`
- `EMAIL_DOMAIN_BLACKLIST` -> `EMAIL_DOMAIN_DENYLIST`
- `EMAIL_DOMAIN_WHITELIST` -> `EMAIL_DOMAIN_ALLOWLIST`

tootctl:

- `tootctl domains purge --whitelist-mode` -> `tootctl domains purge --limited-federation-mode`

Removed badly maintained and no longer relevant .env.production.sample file
2020-06-27 20:20:11 +02:00
Eugen Rochko 85ecbd5527 Fix help text around `tootctl email_domain_blocks` (#14147) 2020-06-26 21:28:40 +02:00
tateisu 29ddd0cdf7 Add tootctl email-domain-blocks (#13589)
* Add tootctl email_domains (block|unblock)

* fix codeclimate issues.

* fix codeclimate issues.

* fix codeclimate issues.

* add list subcommand, remove log_action.

* fix codeclimate issues.

* filter duplicate hostnames,ips before block

* rebase from currnet master branch.
rename email_domains_cli.rb to email_domain_blocks_cli.rb .
rename Mastodon::EmailDomainsCLI to Mastodon::EmailDomainBlocksCLI .
rename command email_domains to email-domain-blocks . (Thor recognizes both of - and _ )
rename subcommand block to add .
rename subcommand unblock to remove .
change the color in list subcommand to while for domain or cyan for childlen.
don't use include() in list subcommand.
suppress console output about succeeded entry.
add console output about count of processed/skipped.
remove capitalization in subcommand description.
remove long_desc in subcommand 'remove'.
remove duplicate where in subcommand 'remove'.

* fix codeclimate issue.
2020-06-25 12:17:10 +02:00
leo60228 230a3b36b6 Add emojis:generate_borders Rake task (#13773)
* Add emojis:generate_borders Rake task

* Address review

* Border all dark emoji

* Combine stroke with filter to reduce artifacting

* Cleanup Camera with Flash

* Add stroke-linejoin="round"

The previous filter and tweaks were effectively a poor imitation of it.
There are no artifacts for any dark emoji now!

* Set stroke-width using property

This fixes old versions of Firefox.

* Store emoji in string instead of array

* Use separate arguments for each path segment

* Remove "background: black;"
2020-06-09 00:12:20 +02:00
ThibG 36f31b5cad Add blurhash to preview cards (#13984)
Fixes #13001
2020-06-05 23:10:41 +02:00
ThibG 8bb1747946 Fix mastodon:setup choking on env variables containing '%' (#13940)
Fix a regression introduced in #13928, caused by TTY::Command building
shell commands by chaining string substitutions.

Ditch TTY::Command and use system instead (both do shell out).
2020-06-03 20:18:52 +02:00
ThibG 96844da338 Fix mastodon:setup not running properly when some environment variables are set (#13928)
When using one of the docker-compose containers, mastodon:setup will use the
existing .env.production rather than the generated one during the setup steps.

This is because dotenv does not overwrite env variables that are alreayd
defined, and the docker-compose.yml file loads the environment variables
from .env.production.
2020-06-02 13:57:21 +02:00
Stanislas acb71734fb Fix `tootctl upgrade storage-schema` S3 ACL (#13768) 2020-05-17 17:27:36 +02:00
Eugen Rochko 7e7613ec9c Fix `tootctl media remove-orphans` choking on unknown files in storage (#13765)
Fix #13762

Catch tootctl interrupt to prevent confusing stacktrace
2020-05-15 18:41:27 +02:00
Eugen Rochko b2488c3d77 Fix `tootctl upgrade storage-schema` misbehaving (#13761)
- Fix not moving original files of custom emojis
- Fix command failing to move any files with S3 storage
- Fix command marking records as upgraded when move failed

Fix #13594
2020-05-15 17:15:24 +02:00
ThibG 74de5ade7d Fix GifReader exceptions (#13760) 2020-05-15 11:38:12 +02:00
Eugen Rochko 5525f6efb7 Bump version to 3.1.4 (#13750) 2020-05-14 07:46:11 +02:00
Takeshi Umeda 27a938101c Fix tootctl media refresh (#13751) 2020-05-14 07:45:52 +02:00
ThibG b7f47014a7 Fix "tootctl media remove-orphans" crashing on “Import” files (#13685)
* Fix "tootctl media remove-orphans" crashing on “Import” files

* Also remove empty directories when removing orphaned media
2020-05-09 21:06:55 +02:00
Hanage999 190768a28d Fix tootctl upgrade storage-schema failing to delete empty directories (#13593) 2020-05-04 13:51:34 +02:00
Lerk 0dd63eae8e Add `tootctl emoji export` (#13534)
* add emoji export command to cli

* fix codeclimate issues

* add error when no matching category was found

* add other suggestions

* exit 1 when no matching category is found

* changes according to suggestions

* 👀

* RubyNein

Y u always autoformat :c
2020-04-27 22:17:49 +02:00
Eugen Rochko adfb22a9dd Add separate cache directory for non-local uploads (#12821) 2020-04-26 23:29:08 +02:00
D Anzorge 945c8e207e Update Twemoji to 12.1.5 (#13021)
* Update Twemoji to 12.1.5

SVGs from Twemoji repo https://github.com/twitter/twemoji

* Update emoji_map.json to Unicode 12
2020-04-19 23:38:31 +02:00
Eugen Rochko ccee0578a4 Fix regression in `tootctl media remove-orphans` (#13405)
Fix #13401
2020-04-06 14:11:47 +02:00
Eugen Rochko 331bc98f25 Fix trying to delete already deleted file when post-processing (#13406)
Fix #13403
2020-04-06 14:11:22 +02:00
Eugen Rochko c7c2a46967 Bump version to 3.1.3 (#13389) 2020-04-05 06:23:46 +02:00
Eugen Rochko 7045cb5d5c Fix `tootctl media remove-orphans` ignoring `PAPERCLIP_ROOT_PATH` (#13375)
Fix #13371
2020-04-02 05:28:51 +02:00
ThibG 5e544da8a4 Fix Paperclip using deprecated URI.escape function (#13320)
Monkey-patch Paperclip to perform URL escaping in a slightly more
appropriate way, and get rid of runtime deprecation warnings.
2020-03-26 15:09:16 +01:00
Eugen Rochko 1cb4645638 Change `tootctl media remove-orphans` to work for all classes (#13316)
Change `tootctl media lookup` to not use an interactive prompt
2020-03-26 01:56:41 +01:00
ThibG 0701e09c85 Fix videos with unsupported colorspace not being transcoded (#13242) 2020-03-10 11:58:40 +01:00
ThibG ccd8ceac47 Fix MP4 (H264 + AAC) video files being needlessly re-encoded (#13239) 2020-03-09 23:15:59 +01:00
Eugen Rochko f459919552 Change local media attachments to perform heavy processing asynchronously (#13210)
Fix #9106
2020-03-08 23:56:18 +01:00
tateisu 03ad395296 Add `--skip-media-remove` option to `tootctl statuses remove` (#13080)
* Add skip_media_remove option to tootctl statuses remove

* Add skip_media_remove option to tootctl statuses remove

Co-authored-by: tateisu <tateisu@juggler.jp>
2020-03-08 16:01:07 +01:00
ThibG 6e937a13a8 Fix installation failing when Redis password contains special characters (#13156)
* Add support for special characters in Redis passwords

Fixes #13154

* Refactor
2020-02-29 03:00:43 +01:00
Eugen Rochko 1a74ffed23 Bump version to 3.1.2 (#13162) 2020-02-27 12:34:07 +01:00
ThibG 5ad9ee92f6 Add `--reset-password` option to `tootctl accounts modify` (#13126) 2020-02-22 01:29:14 +01:00
Marcin Cieślak dc6fbdb6c9 Do not suggest to login into the GitLab database (#13084)
As pointed out on Discourse:

https://discourse.joinmastodon.org/t/obscure-wtf-error-message-running-migrations-for-3-1/2524

The message the administrators were getting was telling them
to log in to the "GitLab database" and also mentions MySQL
2020-02-15 12:52:57 +01:00
Eugen Rochko 28a4b27a41 Bump version to 3.1.1 (#13069) 2020-02-10 02:55:40 +01:00
Eugen Rochko 5cbbe3c3d5 Bump version to 3.1.0 (#13053) 2020-02-09 00:02:08 +01:00
Eugen Rochko 6355996c4b Bump version to 3.1.0rc2 (#12999) 2020-01-28 20:56:32 +01:00
Eugen Rochko 35ea75898c Bump version to 3.1.0rc1 (#12932) 2020-01-24 01:14:51 +01:00
Eugen Rochko f95fd62bac Add announcements (#12662)
* Add announcements

Fix #11006

* Add reactions to announcements

* Add admin UI for announcements

* Add unit tests

* Fix issues

- Add `with_dismissed` param to announcements API
- Fix end date not being formatted when time range is given
- Fix announcement delete causing reactions to send streaming updates
- Fix announcements container growing too wide and mascot too small
- Fix `all_day` being settable when no time range is given
- Change text "Update" to "Announcement"

* Fix scheduler unpublishing announcements before they are due

* Fix filter params not being passed to announcements filter
2020-01-23 22:00:13 +01:00
Jeong Arm cff14409d9 Close registration before self-destruct (#12877) 2020-01-21 18:56:18 +01:00
ThibG 431bd6addc Fix ruby 2.7 warnings about keyword parameters (#12824) 2020-01-11 21:38:02 +01:00
Gomasy b362a77e06 Keep statuses bookmarked by local users in tootctl statuses remove (#12818) 2020-01-11 06:41:37 +01:00
ThibG 5fbadcf9df Add --remote-only option to emoji purge (#12810)
Fixes #12804
2020-01-10 00:10:17 +01:00
ThibG 0b0f358815 Changes to tootctl statuses remove (#11267)
* Fix comments in statuses_cli

The comment for the part of query keeping statuses which have replies was
incorrect, explaining the part of the query kept favourited statuses instead

* Keep statuses favourited by local users in tootctl statuses remove

* Do delete old replies that are not direct interactions with local users

* Skip reblogs of local statuses instead of all reblogs

* Optimize some queries

* Keep old statuses with recent reblogs

* Add option to clear toots from followed accounts too
2019-12-17 13:33:36 +01:00
trwnh 5b127ea614 Fix typo in tootctl statuses remove (#12603) 2019-12-12 19:50:08 +01:00
Eugen Rochko fa15a5e9f2 Fix error handling in `tootctl media remove-orphans` (#12571) 2019-12-09 04:26:00 +01:00
Eugen Rochko 0769ef8277 Add `tootctl media remove-orphans` (#12568) 2019-12-08 15:37:12 +01:00
Yamagishi Kazutoshi b01cb4701e Update ESLint and RuboCop in Code Climate (#12534) 2019-12-02 18:25:43 +01:00
Nathaniel Suchy 8ae8e9268e Add `tootctl media lookup` command (#12283)
* Add a lookup tool to the media cli

* Improved lookup logic

* Clarified wording in the output

* Code style changes

* Code style changes

* Code style changes

* Code style changes

* Add error handling code incase an attachment isn't found

* Code style changes

* Code style changes

* Make requested changes

* Fix styling issues

* Handle other media types

* Remove an inadvertently added log

* Make requested changes

* Make the code safe no matter what the path, S3 or not

* Code style changes

* Code style changes

* Replace select method with Ruby Enumerable grep method
2019-11-04 12:55:20 +01:00
Eugen Rochko 9f94d4ceb3 Fix preloaded JSON-LD context for identity not being used (#12138)
Regression from #11316
2019-10-10 06:48:53 +02:00
Eugen Rochko a4969d4b98 Bump version to 3.0.1 (#12116) 2019-10-09 07:36:57 +02:00
Eugen Rochko 347c82efed Change `tootctl media refresh` to skip already downloaded attachments (#12118) 2019-10-08 05:59:10 +02:00
Eugen Rochko c47d7b8abf Add `tootctl media usage` command (#12115) 2019-10-07 20:04:56 +02:00
Eugen Rochko 7e13c7c710 Fix tootctl not allocating enough database connections for main thread (#12097) 2019-10-07 05:05:02 +02:00
Eugen Rochko 17c2205247 Fix issues with tootctl's parallelization and progress reporting (#12093) 2019-10-07 04:24:05 +02:00
Jeong Arm 9fbb97a0c8 Remove unused option from tootctl accounts cull (#12074) 2019-10-06 04:30:07 +02:00
Eugen Rochko 74ddddd3db Bump version to 3.0.0 (#12000) 2019-10-03 22:44:22 +02:00
Eugen Rochko 6edcffbb9f Bump version to 3.0.0rc3 (#12063) 2019-10-03 04:13:22 +02:00
Eugen Rochko 746fae0bd1 Fix performance of GIF re-encoding (#12057)
* Change animated GIF detection to not shell out to ImageMagick

Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>

* Change video encoding parameters to limit to 10800 video frames

Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>

* Limit GIF image size further

Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>

* Always strip metadata from video files

* Fix code style issues
2019-10-03 01:09:12 +02:00
Takeshi Umeda 38d2632b90 Add parallelization to `tootctl search deploy` (#12051)
* Add parallel gem

* Modify parallel option in tootctl search deploy

* Add paralell option to tootctl search deploy

* Change 1 to false

* Clean up

* Rename --parallel to --processes
2019-10-02 21:50:43 +02:00
Eugen Rochko 36199a7bd6 Change `tootctl domains purge` to accept multiple domains at once (#12046) 2019-10-02 04:49:53 +02:00
Eugen Rochko f77337daaf Fix records not being indexed sometimes (#12024)
It's possible that after commit callbacks were not firing when
exceptions occurred in the process. Also, the default Sidekiq
strategy does not push indexing jobs immediately, which is not
necessary and could be part of the issue too.
2019-10-01 01:19:11 +02:00
Eugen Rochko 7918c98190 Bump version to 3.0.0rc2 (#11999) 2019-09-29 18:50:16 +02:00
Eugen Rochko 3773115066 Fix authentication before 2FA challenge (#11943)
Regression from #11831
2019-09-24 04:35:36 +02:00
Yamagishi Kazutoshi 71ff422997 Add config of multipart threshold for S3 (#11924) 2019-09-23 15:37:45 +02:00
Eugen Rochko c64f143bf7 Bump version to 3.0.0rc1 (#11900) 2019-09-21 20:02:14 +02:00
Eugen Rochko 40d33e780e Bump version to 2.9.3 (#11899) 2019-09-20 22:59:29 +02:00
Eugen Rochko a55f075ab6 Add missing locale file for ga and add rake task to check for it (#11813)
* Add missing locale file for ga and add rake task to check for it

* Update lib/tasks/repo.rake

Co-Authored-By: Yamagishi Kazutoshi <ykzts@desire.sh>

* Fix check-i18n build
2019-09-12 04:58:33 +02:00
Eugen Rochko 4291b74031 Change deletes to preserve soft-deleted statuses in unresolved reports (#11805)
Change all account actions except "none" to resolve all unresolved reports

Refactor `SuspendAccountService` to be more readable
2019-09-11 16:32:44 +02:00
Eugen Rochko 8eab4349b4 Add retry for failed media downloads and `tootctl media refresh` (#11775) 2019-09-10 15:29:12 +02:00
Eugen Rochko 666c1a8a06 Change tootctl to use inline parallelization instead of Sidekiq (#11776)
- Remove --background option
- Add --concurrency(=5) option
- Add progress bars
2019-09-10 13:48:48 +02:00
Eugen Rochko 3f5571301a Add `tootctl cache recount` command (#11597) 2019-08-18 14:55:03 +02:00
Stanislas d67ad75a56 Limit "tootctl accounts follow" to local accounts (#11592)
To (somewhat) limit mass remote follow. Fix #11360
2019-08-17 22:59:40 +02:00
Darius Kazemi ccef31901f Add option to exclude suspended domains/subdomains from tootctl domains crawl (#11454)
* Add "--exclude-suspended" to tootctl domains crawl

This new option ignores any instances suspended server-wide as
well as their associated subdomains. This queries all domain
blocks up front, then runs a regexp on each domain. This improves
performance over what may be the obvious implementation, which is
to ask `DomainBlocks.blocked?(domain)` for each domain -- this
hits the DB many times, slowing things down considerably.

* cleaning up code style

* Compiling regex

* Removing ternary operator
2019-08-03 19:11:09 +02:00
Eugen Rochko 825dc3ca22 Add whitelist mode (#11291) 2019-07-30 11:10:46 +02:00
dependabot-preview[bot] 1cf7bdede4 Bump json-ld-preloaded from 3.0.2 to 3.0.3 (#11316)
* Bump json-ld-preloaded from 3.0.2 to 3.0.3

Bumps [json-ld-preloaded](https://github.com/ruby-rdf/json-ld-preloaded) from 3.0.2 to 3.0.3.
- [Release notes](https://github.com/ruby-rdf/json-ld-preloaded/releases)
- [Commits](https://github.com/ruby-rdf/json-ld-preloaded/compare/3.0.2...3.0.3)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* use json-ld edge
2019-07-28 13:48:43 +02:00
mayaeh 7d13da568e Add `tootctl preview_cards remove` (#11320)
* Add `tootctl preview_cards remove`

* fix code style

* Remove `Scheduler::PreviewCardsCleanupScheduler` file

* fix code style again
Add exclude case where image_file_name is blank

* Added a function to output confirmation if the specified number of days is less than 2 weeks
2019-07-28 13:48:19 +02:00
Clar Fon ea0b3589ff Remove pre from version, add extra suffix variable (#11407) 2019-07-26 07:57:27 +02:00
Eugen Rochko 1c612f24e4 Add categories for custom emojis (#11196)
Fix #7940
2019-06-28 15:54:10 +02:00
Eugen Rochko 2080807d5f Bump version to 2.9.2 (#11152) 2019-06-22 17:28:26 +02:00
Eugen Rochko 9ab22716a1 Bump version to 2.9.1 (#11143) 2019-06-22 01:51:27 +02:00
Eugen Rochko c078bf94a9 Fix converted media being saved with original extension and mime type (#11130) 2019-06-20 10:52:36 +02:00
Eugen Rochko 500276ce96 Bump version to 2.9.0 (#11074) 2019-06-13 20:19:21 +02:00
Eugen Rochko a388bc818b Bump version to 2.9.0rc2 (#11070) 2019-06-13 00:43:59 +02:00
Eugen Rochko 13f112c116 Bump version to 2.9.0rc1 (#11004) 2019-06-09 15:53:08 +02:00
Darius Kazemi 66ec27e0c8 Specify gzip required in tootctl emoji help (#11000) 2019-06-08 12:43:11 -04:00
Eugen Rochko 48abfb938f Forward port version bumps to 2.8.3 and 2.8.4 (#10819)
* Bump version to 2.8.3

* Bump version to 2.8.4
2019-05-24 15:57:31 +02:00
ThibG f1a3135809 Record account suspend/silence time and keep track of domain blocks (#10660)
* Record account suspend/silence time and keep track of domain blocks

* Also unblock users who were suspended/silenced before dates were recorded

* Add tests

* Keep track of suspending date for users suspended through the CLI

* Show accurate number of accounts that would be affected by unsuspending an instance

* Change migration to set silenced_at and suspended_at

* Revert "Also unblock users who were suspended/silenced before dates were recorded"

This reverts commit a015c65d2d1e28c7b7cfab8b3f8cd5fb48b8b71c.

* Switch from using suspended and silenced to suspended_at and silenced_at

* Add post-deployment migration script to remove `suspended` and `silenced` columns

* Use Account#silence! and Account#suspend! instead of updating the underlying property

* Add silenced_at and suspended_at migration to post-migration

* Change account fabricator to translate suspended and silenced attributes

* Minor fixes

* Make unblocking domains always retroactive
2019-05-14 19:05:02 +02:00
Jeong Arm 8d4c489681 Remove custom emojis on "tootctl domains purge" (#10721)
* Remove custom emojis on domains purge

* Change message "Removing" to "Removed"
2019-05-09 22:03:20 +02:00
Eugen Rochko 0979e7f2f5 Bump version to 2.8.2 (#10703) 2019-05-05 17:14:15 +02:00
Ushitora Anqou 3feece700b Add SOURCE_TAG to show source repository's tag (#10698) 2019-05-04 17:39:17 +02:00
Eugen Rochko a33bd6ad98 Add `tootctl cache clear` (#10689) 2019-05-04 01:02:57 +02:00
Eugen Rochko 52eca0c4b5 Bump version to 2.8.1 (#10687) 2019-05-04 00:31:06 +02:00
Eugen Rochko cc9851a479 Fix accounts created through tootctl not being always pre-approved (#10684)
Add `--approve` option to `tootctl accounts modify`
2019-05-03 20:49:27 +02:00
Eugen Rochko a6d2fe7165 Add blurhash (#10630)
* Add blurhash

* Use fallback color for spoiler when blurhash missing

* Federate the blurhash and accept it as long as it's at most 5x5

* Display unknown media attachments as blurhash placeholders

* Improve style of embed actions and spoiler button

* Change blurhash resolution from 3x3 to 4x4

* Improve dependency definitions

* Fix code style issues
2019-04-27 03:24:09 +02:00
Sho Kusano 1af37e09f3 Configrationable repository url (#10600)
* config: Add GITHUB_REPOSITORY for repository name

* config: Add SOURCE_BASE_URL for repository url

* Show source_url and repository name on getting started
2019-04-21 04:41:34 +02:00
Eugen Rochko 373fa21cf7 Bump version to 2.8.0 (#10550) 2019-04-10 23:32:08 +02:00
Eugen Rochko 01dc793a94 Bump version to 2.8.0rc3 (#10535) 2019-04-10 04:25:09 +02:00
Eugen Rochko ae6c9b4f10 Bump version to 2.8.0rc2 (#10501) 2019-04-09 05:04:11 +02:00
Takeshi Umeda 01f46add16 Add `tootctl accounts reset-relationships` (#10483)
* Add `tootctl accounts reset`

* Rename reset to reset-relationships

* Improve command description
2019-04-08 07:46:55 +02:00
Eugen Rochko 07c6a0edc2 Fix missing long description on `tootctl statuses remove` (#10482) 2019-04-06 04:47:26 +02:00
Eugen Rochko c455cba4d8 Add `tootctl emoji purge` (#10481)
Fix #10441
2019-04-06 04:47:16 +02:00
Eugen Rochko a9c29f6e15 Add `tootctl accounts approve` (#10480) 2019-04-06 04:47:05 +02:00
Ben Lubar e9ef322e14 Fix `tootctl accounts cull` (#10460)
* List the actual accounts that would have been culled during a dry run.

Otherwise, the dry run mode is basically useless.

* Prevent unreachable domains from inheriting the previous status code.

* Update CHANGELOG.md for #10460.
2019-04-04 16:46:27 +02:00
Jeong Arm dbabfec2ea Simplify culling and fix race condition (#10440) 2019-04-01 19:17:21 +02:00
Eugen Rochko 2dc1f643d9 Bump version to 2.8.0rc1 (#10340)
Update AUTHORS.md
Update CHANGELOG.md
2019-03-30 02:13:42 +01:00
Eugen Rochko 3a9abe39b2 Add `tootctl self-destruct` (#10367)
Fix #10305
2019-03-28 17:56:25 +01:00
Eugen Rochko b6bf8e24be Add `tootctl search deploy` to avoid ugly rake task syntax (#10403) 2019-03-28 02:16:11 +01:00
Eugen Rochko 73fb7bfa0f Admission-based registrations mode (#10250)
Fix #6856
Fix #6951
2019-03-14 05:28:30 +01:00
Eugen Rochko e7bc85a91a Add `tootctl statuses remove` to sweep unreferenced statuses (#10063)
Query by @tateisu

Fix #1554
2019-03-11 13:45:17 +01:00
Eugen Rochko 657436c5db Bump version to 2.7.4 (#10179) 2019-03-05 18:22:46 +01:00
Eugen Rochko c87532494d Bump version to 2.7.3 (#10102) 2019-02-23 22:20:20 +01:00
Eugen Rochko db31bef536 Bump version to 2.7.2 (#10067) 2019-02-17 20:24:12 +01:00
Eugen Rochko 0848f9b66c Bump version to 2.7.1 (#9932) 2019-01-28 04:24:12 +01:00
Stanislas 92801eba9e Fix SUPERUSER postgres command (#9877) 2019-01-27 23:56:07 +01:00
Eugen Rochko a338ea38bd Fix `tootctl accounts delete` not deleting user record as well (#9874) 2019-01-20 12:30:25 +01:00
Eugen Rochko b4fece9327 Bump version to 2.7.0 2019-01-20 00:30:53 +01:00
Eugen Rochko e73151320d Add translators to AUTHORS.md (#9865) 2019-01-19 22:01:12 +01:00
Eugen Rochko f149ffbcbe Change CHANGELOG.md references from pull requests to authors (#9864)
Add `rake repo:changelog` to do that automatically
2019-01-19 16:25:20 +01:00
Eugen Rochko 747bfbe950 Bump to 2.7.0rc3 2019-01-19 00:01:27 +01:00
Eugen Rochko 2c118c9229 Bump version to 2.7.0rc2 (#9836) 2019-01-18 22:55:01 +01:00
Eugen Rochko 73fbbd48cd Add `tootctl --version` (#9835) 2019-01-17 21:32:55 +01:00
Eugen Rochko 7a4958b028 Fix `tootctl domains crawl` with JSON format output crash (#9820)
Fix #9817
2019-01-16 15:47:55 +01:00
Eugen Rochko 1c1ca9d155 Add `tootctl domains crawl` (#9809) 2019-01-15 09:24:35 +01:00
Eugen Rochko 0963a4400b Bump version to 2.7.0rc1 (#9718) 2019-01-08 14:10:54 +01:00
Eugen Rochko 611a9ff3fe Add tootctl accounts follow ACCT (#9414)
Fix #9369
2019-01-01 16:24:26 +01:00
Eugen Rochko bbf9f4f93b Add REST API for creating an account (#9572)
* Add REST API for creating an account

The method is available to apps with a token obtained via the client
credentials grant. It creates a user and account records, as well as
an access token for the app that initiated the request. The user is
unconfirmed, and an e-mail is sent as usual.

The method returns the access token, which the app should save for
later. The REST API is not available to users with unconfirmed
accounts, so the app must be smart to wait for the user to click a
link in their e-mail inbox.

The method is rate-limited by IP to 5 requests per 30 minutes.

* Redirect users back to app from confirmation if they were created with an app

* Add tests

* Return 403 on the method if registrations are not open

* Require agreement param to be true in the API when creating an account
2018-12-24 19:12:38 +01:00
Eugen Rochko 36a5585cac Update emoji codepoint mappings to v11.0 (#9618) 2018-12-24 19:12:00 +01:00
Eugen Rochko 652595d802 Fix tootctl accounts rotate not updating public keys (#9556)
This allowed you to brick your system when running that command, because the accounts would continue to advertise the old public key, but sign things with the new one
2018-12-18 01:22:29 +01:00
Eugen Rochko 626a544d08 Improve account suspension speed and completeness (#9290)
- Some associations were missing from the clean-up
- Some attributes were not reset on suspension
- Skip federation and streaming deletes when purging a dead domain
- Move account association definitions to concern
2018-12-03 01:32:08 +01:00
Eugen Rochko 9b85750348 Bump version to 2.6.5 (#9413) 2018-12-02 17:41:38 +01:00
ThibG 4813053e97 Preload common JSON-LD contexts (#9412)
Fixes #9411
2018-12-02 16:46:13 +01:00
Eugen Rochko ecc22893f6 Bump version to 2.6.4 (#9402) 2018-11-30 21:38:09 +01:00
Eugen Rochko ab3f2aba04 Bump version to 2.6.3 (#9391) 2018-11-30 14:20:15 +01:00
Eugen Rochko 163c5e6893 Bump version to 2.6.2 (#9370) 2018-11-27 13:56:47 +01:00
Renato "Lond" Cerqueira 55aa92b893 Touch account on successful response, change char shown when culled (#9293)
Just the color is not enough change since not everyone uses colored
terminals.
Touching the account makes it so that the account is not in the
threshold window in case of running again
2018-11-20 22:25:32 +01:00
mayaeh 214aab8715 Fix "tootctl media remove" can't count the file size (#9288)
* Fixed an issue where "tootctl media remove" can not count the file size.

* Fixed the problem pointed out by codeclimate.
2018-11-16 09:47:40 +01:00
Eugen Rochko eed84cacba Fix nil error regression from #9229 in tootctl media remove (#9239)
Fix #9237
2018-11-08 21:55:59 +01:00
Eugen Rochko a178e66988 Display amount of freed disk space in tootctl media remove (#9229)
* Display amount of freed disk space in tootctl media remove

Fix #9213

* Fix code style issue
2018-11-08 21:06:26 +01:00
Eugen Rochko 56e256b686 Bump version to 2.6.1 (#9172) 2018-10-31 01:04:45 +01:00
Eugen Rochko 15fd0fb2cf Bump version to 2.6.0 (#9149)
* Bump version to 2.6.0

* Update CHANGELOG.md
2018-10-30 22:06:59 +01:00
Eugen Rochko 4c8ac98fa0 Bump version to 2.6.0rc4 (#9164)
* Bump version to 2.6.0rc4

* Update CHANGELOG.md
2018-10-30 17:37:49 +01:00
Eugen Rochko e6c91483fb Bump version to 2.6.0rc3 (#9142)
* Bump version to 2.6.0rc3

* Update CHANGELOG.md
2018-10-29 14:15:54 +01:00
Eugen Rochko f435af86e7 Do not remove "dead" domains in tootctl accounts cull (#9108)
Leave `tootctl accounts cull` to simply check removed accounts from
live domains, and skip temporarily unavailable domains, while listing
them in the final output for further action.

Add `tootctl domains purge DOMAIN` to be able to purge a domain from
that list manually
2018-10-27 22:56:16 +02:00
Sascha 94b16add64 cli: set exit_on_failure for all CLI classes (#9094) 2018-10-25 16:05:33 +02:00
Eugen Rochko 9035c156c2 Bump version to 2.6.0rc2 (#9087)
* Bump version to 2.6.0rc2

* Update CHANGELOG.md
2018-10-25 04:42:25 +02:00
Eugen Rochko 7f1b063420 Fix cull tripping on nil in last_webfingered_at (#9051)
Fix #8741
2018-10-22 16:58:08 +02:00
Jeong Arm e1a7e7b9cc Fix tootctl cull on dead servers (#9041)
* Delete first 9 accounts on dead servers

* Clean up code by moving dead server culling to the end
2018-10-21 22:52:27 +02:00
Jeong Arm 2f0318d360 Handle if username is not found on tootctl feeds build (#9040) 2018-10-21 16:42:22 +02:00
Eugen Rochko bb62322411 Bump version to 2.6.0rc1 (#9025)
* Bump version to 2.6.0rc1

* Update AUTHORS.md

* Update CHANGELOG.md
2018-10-21 00:49:36 +02:00
Eugen Rochko 6cd4eaaee7 Bump version to 2.5.2 (#8960) 2018-10-12 00:24:09 +02:00
Eugen Rochko a441df3e19 Bump version to 2.5.1 (#8953) 2018-10-11 02:29:32 +02:00
ashleyhull-versent 00d31a292c rubocop issues - Cleaning up (#8912)
* cleanup pass

* undo mistakes

* fixed.

* revert
2018-10-08 04:50:11 +02:00
Eugen Rochko 1a1160ee24 Add tootctl settings registrations open (#8829) 2018-10-04 16:05:54 +02:00
Eugen Rochko 20beab56e4 Fix tootctl accounts reattaching not unsuspending deleted account (#8812) 2018-09-29 18:40:40 +09:00
Eugen Rochko 6dd97b97be Add tootctl accounts backup (#8811) 2018-09-28 03:34:24 +02:00
ThibG 17ff4b960f Unconditionally re-encode locally-uploaded images to strip metadata (#8714)
This strips metadata on file upload by re-encoding the files, at the cost
of possible slight image quality decrease and processing resources.
2018-09-17 20:24:46 +02:00
Eugen Rochko 26d73847cb Move more tasks to tootctl (#8675)
* Move more tasks to tootctl

- tootctl feeds build
- tootctl feeds clear
- tootctl accounts refresh

Clean up exit codes and help messages

* Move user modifying to tootctl

* Improve user modification through CLI, rename commands

add -> create
mod -> modify
del -> delete

To remove ambiguity

* Fix code style issues

* Fix not being able to unset admin/mod role
2018-09-14 17:42:22 +02:00
luzpaz 1bce70d3c7 Misc. typos (#8694)
Found via `codespell -q 3 --skip="./app/javascript/mastodon/locales,./config/locales"`
2018-09-14 00:53:09 +02:00
rinsuki 14604c384f using mailto scheme in AUTHORS.md (#8663) 2018-09-10 02:01:03 +02:00
nightpool b778e40d7d Add rake task for generating AUTHORS.md (#8661)
* add rake task for generating AUTHORS.md

* update AUTHORS.md

* rubocop
2018-09-10 01:31:42 +02:00
Eugen Rochko 2aee3c9f23 tootctl accounts [add|del|cull] (#8642)
* CLI interface for creating/deleting local users

- tootctl accounts add USERNAME
- tootctl accounts del USERNAME

* Add CLI interface for culling remote users that no longer exist

- tootctl accounts cull
2018-09-09 13:33:36 +02:00
Eugen Rochko e3fdf02943 Bump version to 2.5.0 2018-09-02 22:32:27 +02:00
Eugen Rochko 894e01163b Bump version to 2.5.0rc2 2018-09-01 03:37:43 +02:00
Eugen Rochko 32d3152852 Fix wrong string being used on login failure when using LDAP (#8534)
Fix #8527
2018-09-01 03:35:37 +02:00
mayaeh 6823e24a9d Add --verbose and --dry-run option to tootctl media remove (#8519)
* Add --verbose and --dry-run options to CLI interface for removing remote media.

* Fix coding style problem.
2018-08-31 03:46:13 +02:00
mayaeh 2adac68cc7 Fix CLI interface for removing remote media (#8506)
* Fix that can't delete media files even if "tootctl media remove" execute when "--background" not attached.

* Revert

This reverts commit 5aa7e09645b27bae38a26030148b23e553ee2662.

* Change to obtain and pass all columns when "--background" option is false.
2018-08-29 18:35:09 +02:00
Eugen Rochko 1d319c531e Add CLI task for rotating keys (#8466)
* If an Update is signed with known key, skip re-following procedure

Because it means the remote actor did *not* lose their database

* Add CLI method for rotating keys

    bin/tootctl accounts rotate [USERNAME]

Generates a new RSA key per account and sends out an Update activity
signed with the old key.

* Key rotation: Space out Update fan-outs every 5 minutes per 1000 accounts

* Skip suspended accounts in key rotation
2018-08-26 20:21:03 +02:00
Quint Guvernator 476ba6e4f0 Fix low-hanging rubocop gripes (#8458)
* rubocop: quit being so picky

* rubocop: miscellany

* rubocop: prefer present to blank
2018-08-26 19:22:46 +02:00
Eugen Rochko ee4af13f51 Fix NUM_DAYS in rake mastodon:media:remove_remote and print warning (#8462)
Remove old deprecated rake tasks
2018-08-26 17:19:47 +02:00
Eugen Rochko 7271a40d09 Add CLI interface for importing custom emoji (#8437)
bin/tootctl emoji import PATH_TO_TAR

Fix #8435
2018-08-26 16:53:06 +02:00
M Somerville 4b27569841 Rename S3_CLOUDFRONT_HOST to S3_ALIAS_HOST. (#8423)
Still check for S3_CLOUDFRONT_HOST for existing installs.
2018-08-25 13:27:08 +02:00
Eugen Rochko e6e845f74d Add improved CLI interface for removing remote media (#8411)
./bin/tootctl media remove --days 7 --background

Make the old rake task point to it
2018-08-25 13:25:39 +02:00
Eugen Rochko fc45d4a096 Bump version to 2.5.0rc1 (#8356) 2018-08-23 23:30:57 +02:00
Eugen Rochko fe88a4f42c Get rid of all batch order warnings (#8334) 2018-08-21 12:25:50 +02:00
Immae cbaabe0215 Add ldap search filter (#8151) 2018-08-15 18:12:44 +02:00
Eugen Rochko 98184a889e Add post-deployment migration system (#8182)
Adopted from GitLab CE. Generate new migration with:

    rails g post_deployment_migration name_of_migration_here

By default they are run together with db:migrate. To not run them,
the env variable SKIP_POST_DEPLOYMENT_MIGRATIONS must be set

Code by Yorick Peterse <yorickpeterse@gmail.com>, see also:

83c8241160
2018-08-13 13:40:01 +02:00
ThibG 40c7bd7cdd Add some feedback to maintenance rake tasks (#8173) 2018-08-11 22:02:55 +02:00
Eugen Rochko 13df3889f0 Redesign public profiles and toots (#8068) 2018-07-28 19:25:33 +02:00
Eugen Rochko 91aadec929 Resize images by area instead of fixed dimensions (#8083)
To improve the way super tall or super ride images are treated, the
numbers remain the same, 1280x1280 and 400x400, but if an image
is less in one dimension than the other, the other can become larger

Thanks to @WAHa_06x36@mastodon.social for the tip
2018-07-28 03:33:00 +02:00
Eugen Rochko 9ac778ba1e Fix static GIFs being saved as empty files when using local storage (#8012)
Fix #7997
Fix #6237
2018-07-14 03:56:52 +02:00
Eugen Rochko 54a78d1837 Bump version to 2.4.3 2018-07-11 11:21:27 +02:00
Eugen Rochko ea88974c04 Bump version to 2.4.3rc3 2018-07-11 01:31:23 +02:00
Eugen Rochko 52fd091fe7 Bump version to 2.4.3rc2 2018-07-07 22:23:52 +02:00
Eugen Rochko 47fd3f260c Bump version to 2.4.3rc1 2018-07-05 21:08:03 +02:00
Eugen Rochko 5db8a0ba36 Bump version to 2.4.2 2018-06-19 01:45:00 +02:00
Eugen Rochko d2a1e2e850 Bump version to 2.4.2rc3 2018-06-19 00:19:54 +02:00
Eugen Rochko 8b2ab128a2 Bump version to 2.4.2rc2 2018-06-18 23:01:52 +02:00
Eugen Rochko 5bcb18127c Bump version to 2.4.2rc1 2018-06-15 20:38:41 +02:00
Eugen Rochko daf9bf9f12 Bump version to 2.4.1 2018-06-10 17:20:56 +02:00
Eugen Rochko e23ccfa059 Bump version to 2.4.1rc4 2018-06-10 00:39:56 +02:00
Eugen Rochko e28866b313 Bump version to 2.4.1rc3 2018-06-06 21:27:06 +02:00
Eugen Rochko 36528f28a5 Bump version to 2.4.1rc2 2018-06-05 11:53:30 +00:00
Eugen Rochko 797c72329f Fix missing column in select in mastodon:feeds:build task (#7720) 2018-06-04 04:21:15 +02:00
Eugen Rochko 49a0a3a8a2 Speed up some rake tasks by moving execution to Sidekiq (#7678)
* Speed up some rake tasks by moving execution to Sidekiq

mastodon:media:remove_silenced
mastodon:media:remove_remote
mastodon:media:redownload_avatars
mastodon:feeds:build

* Fix code style issue
2018-05-30 21:09:30 +02:00
Eugen Rochko eebe7fb2d5 Bump version to 2.4.1rc1 2018-05-30 19:16:41 +02:00
Eugen Rochko 8205147d04 Bump version to 2.4.0 2018-05-22 19:39:35 +02:00
Eugen Rochko bbdce87b80 Bump version to 2.4.0rc5 2018-05-21 16:05:31 +02:00
Eugen Rochko 604eae1677 Bump version to 2.4.0rc4 2018-05-19 18:38:10 +02:00
MIYAGI Hikaru 7dbf83d6b8 User agent for WebFinger (#7531)
* User agent for WebFinger

* local_domain → web_domain

* 'http' is away accidentally...
2018-05-18 01:47:22 +02:00
Eugen Rochko dceab1af01 Bump version to 2.4.0rc3 2018-05-14 17:46:44 +02:00
Eugen Rochko 759b2b9c99 Bump version to 2.4.0rc2 2018-05-14 14:49:38 +02:00
Eugen Rochko 0dcb01855c Bump version to 2.4.0rc1 (#7283) 2018-05-06 11:50:00 +02:00
ThibG f2ecedaab5 Fixes/do not override timestamps (#7336)
* Revert "Fixes/do not override timestamps (#7331)"

This reverts commit 9c1e1005f4.

* Document Snowflake ID corner-case a bit more

Snowflake IDs are used for two purposes: making object identifiers harder to
guess and ensuring they are in chronological order. For this reason, they
are based on the `created_at` attribute of the object.

Unfortunately, inserting items with older snowflakes IDs will break the
assumption of consumers of the paging APIs that new items will always have
a greater identifier than the last seen one.

* Add `override_timestamps` virtual attribute to not correlate snowflake ID with created_at
2018-05-03 23:02:46 +02:00
Eugen Rochko ca1c696dbd Slightly reduce RAM usage (#7301)
* No need to re-require sidekiq plugins, they are required via Gemfile

* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile

* Replace ruby-oembed with FetchOEmbedService

Reduce startup by 45382 allocated objects

* Remove preloaded JSON-LD in favour of caching HTTP responses

Reduce boot RAM by about 6 MiB

* Fix tests

* Fix test suite by stubbing out JSON-LD contexts
2018-05-02 18:58:48 +02:00
Eugen Rochko 0370c0f4d6 Add support for separate Redis for cache (#7272)
* Add support for separate Redis for cache

CACHE_REDIS_URL to allow using a different Redis server for cache
purposes, with cache-specific configuration such as key eviction

* Fix code style issues
2018-04-27 01:37:59 +02:00
Alejandro Martinez Ruiz 20fcb1f5ae tasks/mastodon: fix prompt for Redis password (#7241) 2018-04-23 16:03:58 +02:00
David Baucum 259e3aa7bf Ability to specify Redis passwd on mastodon:setup (#7222)
Closes #7221
2018-04-22 11:49:16 +02:00
Yamagishi Kazutoshi 3aee1034e5 Add extract_foreign_key_action to Mastodon::MigrationHelpers (#7195) 2018-04-20 12:21:28 +02:00
Eugen Rochko ec74c3d7da Do not default SMTP verify mode to "peer", default to "none" (#6996) 2018-04-02 19:19:51 +02:00
Eugen Rochko ebba6fb7f5 Fix UniqueUsernameValidator comparison (#6926)
Comparison was downcasing only one side, therefore if previously
existing account had a non-lowercase spelling, it would be ignored
when checking for duplicates.

New rake task `mastodon:maintenance:find_duplicate_usernames` will
help find constraint violations that might have occured from the
presence of this bug.

Bump version to 2.3.3
2018-03-27 04:33:57 +02:00
Akihiko Odaki 11c19f6cc9 Close http connection in perform method of Request class (#6889)
HTTP connections must be explicitly closed in many cases, and letting
perform method close connections makes its callers less redundant and
prevent them from forgetting to close connections.
2018-03-24 12:49:54 +01:00
Eugen Rochko 3939ca8a33 Bump version to 2.3.2 2018-03-22 14:13:46 +01:00
Eugen Rochko f4ef86a334 Bump version to 2.3.2rc5 2018-03-22 11:31:52 +01:00
Eugen Rochko 8ccf42c149 Bump version to 2.3.2rc4 2018-03-20 23:49:24 +01:00
Eugen Rochko 59b3b38b0e Add LDAP_TLS_NO_VERIFY option, don't require LDAP_ENABLED outside .env (#6845)
Fix #6816, fix #6790
2018-03-20 19:41:51 +01:00
Eugen Rochko fd86b58568 Bump version to 2.3.2rc3 2018-03-19 12:20:57 +01:00
Eugen Rochko a9538ac293 Bump version to 2.3.2rc2 2018-03-17 15:28:52 +01:00
Eugen Rochko 64d5c8a512 Bump version to 2.3.2rc1 2018-03-17 14:07:00 +01:00
Akihiko Odaki 7b247b15f2 Correct the reference to user's password in mastodon:add_user task (#6800) 2018-03-17 13:27:19 +01:00
Eugen Rochko 61ad11683d Bump version to 2.3.1 2018-03-13 15:50:15 +01:00
Eugen Rochko 3bf841c74e Bump version to 2.3.1rc3 2018-03-13 09:21:21 +01:00
Ushitora Anqou 44917a10ad Detailed SMTP setup (#6759)
* add detailed SMTP settings setup in mastodon:setup

* add localhost SMTP settings setup in mastodon:setup

* SMTP settings setup should exit after successful delivery of test mail
2018-03-12 21:41:26 +01:00
艮 鮟鱇 aa3b1144d1 set SAFETY_ASSURED=1 of db:setup in mastodon:setup (#6758) 2018-03-12 16:21:48 +01:00
Eugen Rochko 9350e691cc Bump version to 2.3.1rc2 2018-03-12 12:48:49 +01:00
Eugen Rochko b744cfee18 Bump version to 2.3.1rc1 2018-03-11 15:13:13 +01:00
Eugen Rochko f08bc1f188 Bump version to 2.3.0 2018-03-09 12:59:58 +01:00
Eugen Rochko 512e42fed8 When inside Docker, output saved configuration during mastodon:setup (#6711) 2018-03-09 11:52:18 +01:00
Eugen Rochko 38e93a9541 Bump version to 2.3.0rc3 2018-03-08 11:19:02 +01:00
Eugen Rochko f92176899e After blocking domain with reject_media, invalidate cache (#6679)
Media attachments are part of the association cache of statuses,
since they are presumed to be immutable. Unless this cache is
cleared manually, the statuses will continue to look like they
have media embedded.
2018-03-08 06:59:42 +01:00
Eugen Rochko 5b2d4c84d9 Bump version to 2.3.0rc2 2018-03-07 12:06:23 +01:00
Eugen Rochko ba865c44d8 Fix #3807: Increase avatars to 400x400 max (#6651)
But do not upscale when they are smaller
2018-03-06 06:29:01 +01:00
Akihiko Odaki 0016fd40e7 Do not fetch environment variables to determine default locale (#6618)
The default locale is now set by config.
2018-03-04 10:00:46 +01:00
Thomas Leister 225d270134 Fixes #6584 (#6585) 2018-03-01 23:30:06 +01:00
Eugen Rochko 4bd1403368 Bump version to 2.3.0rc1 2018-03-01 20:50:23 +01:00
Eugen Rochko 9721b7746a Fix #942: Seamless LDAP login (#6556) 2018-02-28 19:04:53 +01:00
Paul Woolcock fee184c0fd Some images can cause `convert` to fail, which crashes this whole task (#6565)
* Some images can cause `convert` to fail, which crashes this whole task

* Add more specific exception
2018-02-26 22:01:49 +01:00
Eugen Rochko f35356609b Ensure the app does not even start if OTP_SECRET is not set (#6557)
* Ensure the app does not even start if OTP_SECRET is not set

* Remove PAPERCLIP_SECRET (it's not used by anything, actually)

Imports are for internal consumption and the url option isn't even
used correctly, so we can remove the hash stuff from them
2018-02-26 01:31:44 +01:00
Eugen Rochko c9ed272a4a Fix #6525: Make sure file is opened in LazyThumbnail processor (#6529) 2018-02-22 00:28:19 +01:00
Eugen Rochko f6884555d7 Account archive download (#6460)
* Fix #201: Account archive download

* Export actor and private key in the archive

* Optimize BackupService

- Add conversation to cached associations of status, because
  somehow it was forgotten and is source of N+1 queries
- Explicitly call GC between batches of records being fetched
  (Model class allocations are the worst offender)
- Stream media files into the tar in 1MB chunks
  (Do not allocate media file (up to 8MB) as string into memory)
- Use #bytesize instead of #size to calculate file size for JSON
  (Fix FileOverflow error)
- Segment media into subfolders by status ID because apparently
  GIF-to-MP4 media are all named "media.mp4" for some reason

* Keep uniquely generated filename in Paperclip::GifTranscoder

* Ensure dumped files do not overwrite each other by maintaing directory partitions

* Give tar archives a good name

* Add scheduler to remove week-old backups

* Fix code style issue
2018-02-21 23:21:32 +01:00
takayamaki 2b3d62a8a5 fix purge_removed_accounts task should suspend account before delete it (#6521) 2018-02-21 16:30:46 +01:00
Eugen Rochko c34ff9fb82 Fix avatar and header issues by using custom geometry detector (#6515)
* Fix avatar and header issues by using custom geometry detector

Revert a part of #6508. The file passed to dynamic styles method
was not actually a file, but an instance of Paperclip::Attachment,
which broke all styles by always returning {} from the method.

One problem with GIF avatars was that Paperclip::GeometryDetector
reported wrong dimensions for them, e.g. 120x120 GIF avatar would
for some reason be detected as 120x53. By writing our own geometry
parser, we can use FastImage, which also happens to be faster than
ImageMagick, to detect image dimensions, which are also correct.

Unfortunately, this PR does not implement skipping a `convert`
entirely if the dimensions are already correct, as I found no easy
way to write that behaviour into Paperclip without rewriting the
Paperclip::Thumbnail class.

* Only invoke convert if dimension or format needs to be changed
2018-02-21 03:40:12 +01:00
Eugen Rochko e8e6786d6f Interactive `rake mastodon:setup` task (#6451)
* Add better CLI prompt

* Add rake mastodon:setup interactive wizard

* Test db/redis/smtp configurations and add admin user at the end

* Test database connection even when database does not exist yet
2018-02-11 18:40:57 +01:00
Akihiko Odaki 5892127534 Require environment for generate_static_pages (#6420)
It is required for ApplicationController.
2018-02-03 18:41:01 +01:00
Eugen Rochko 9d2cb4d12d Bump version to 2.2.0 2018-01-29 01:12:36 +01:00
Eugen Rochko 9c212f0f28 ASSET_HOST is wrong env variable. Fix to CDN_HOST (#6372) 2018-01-29 01:06:39 +01:00
Eugen Rochko de667c2a95 Bump to 2.2.0rc2 2018-01-24 04:16:30 +01:00
Eugen Rochko 8f60c22ba4 Fix #6311: Replace relative URLs in CSS only for Premailer (#6335) 2018-01-24 02:57:14 +01:00
Eugen Rochko b9794f1a49 Bump version to 2.2.0rc1 2018-01-20 16:27:19 +01:00
Eugen Rochko 896cadceed Improve HTML e-mails based on Litmus tests (#6301)
* Use PNG images in HTML e-mails

* Make webpack use URLs with host so fonts load inside HTML e-mails

Convert this back to a relative URL in the premailer CSS loader
since local requests are quicker

* Improve responsive design

* Add missing PNG icon
2018-01-20 01:32:21 +01:00
Eugen Rochko 92df0bba59 Fix regeneration marker not expiring (#6290)
* Fix regeneration key not getting expired

* Add rake task to remove old regeneration markers
2018-01-18 20:29:56 +01:00
Eugen Rochko 9613c3238c HTML e-mails for UserMailer (#6256)
- premailer gem to turn CSS into inline styles automatically
- rework UserMailer templates
- reword UserMailer templates
2018-01-16 03:29:11 +01:00
Eugen Rochko 67ee638c28 Move e-mail digest task to sidekiq, reduce workload, improve hint (#6252) 2018-01-15 04:34:28 +01:00
Eugen Rochko 87dd894cf4 Bump version to 2.1.3 2018-01-08 19:15:31 +01:00
Eugen Rochko 76268627c2 Bump version to 2.1.2 2018-01-05 04:52:06 +01:00
Eugen Rochko a0273c2c5e Bump version to 2.1.1 (#6164) 2018-01-04 16:40:26 +01:00
Eugen Rochko 50f2d69780 Add rake task to check and purge accounts that are missing in origin (#6085)
* Add rake task to check and purge accounts that are missing in origin

* Add progress bar and --force options to mastodon:maintenance:purge_removed_accounts
2017-12-24 16:14:33 +01:00
Daigo 3 Dango 64ae134e7b Remove period from the version number (#6039)
2.1.0. -> 2.1.0
2017-12-15 21:38:25 +01:00
Eugen Rochko b23c9ab7de Bump version to 2.1.0 🎆 2017-12-15 19:14:57 +01:00
Eugen Rochko 1079099d2b Bump version to 2.1.0rc6 2017-12-15 02:00:28 +01:00
Eugen Rochko d4849787c5 Bump version to 2.1.0rc5 2017-12-13 20:53:39 +01:00
Eugen Rochko cae49f11c7 Bump version to 2.1.0rc4 2017-12-13 12:45:12 +01:00
Eugen Rochko a6fdeedb47 Bump version to 2.1.0rc3 2017-12-09 02:42:59 +01:00
Eugen Rochko 3126869f5a Bump version to 2.1.0rc2 2017-12-08 01:47:08 +01:00
Akihiko Odaki ce3989fc6a Add embed_url to preview cards (#5775) 2017-12-07 03:37:43 +01:00
Eugen Rochko 11a9900269 Bump version to 2.1.0rc1 (#5834) 2017-12-06 16:13:32 +01:00
Yamagishi Kazutoshi fba46b6072 Using double splat operator (#5859) 2017-12-06 11:41:57 +01:00
ysksn 08cb63c360 Remove empty strings (#5732) 2017-11-17 10:52:30 +09:00
Eugen Rochko 97178b6d86 Add moderator role and add pundit policies for admin actions (#5635)
* Add moderator role and add pundit policies for admin actions

* Add rake task for turning user into mod and revoking it again

* Fix handling of unauthorized exception

* Deliver new report e-mails to staff, not just admins

* Add promote/demote to admin UI, hide some actions conditionally

* Fix unused i18n
2017-11-11 20:23:33 +01:00
Naoki Kosaka bc20bea20e In remove_remote, exclude removed media attachments. (#5626) 2017-11-08 15:19:22 +01:00
unarist 4472c5723a Fix Cocaine::ExitStatusError when upload small non-animated GIF (#5489)
Looks like copied tempfile need to be flushed before further processing. This issue won't happen if the uploaded file has enough file size.
2017-10-26 22:48:35 +09:00
Eugen Rochko b222a9d5ee Bump version to 2.0.0 🐘 2017-10-18 13:53:56 +02:00
Eugen Rochko cb48c98e02 Bump version to 2.0.0rc4 2017-10-17 23:16:35 +02:00
Eugen Rochko 7603119d3a Bump version to 2.0.0rc3 2017-10-16 16:29:00 +02:00
aschmitz 93e3eaf95d Fix #5329 (#5332)
This fixes #5329, which occurred when using MigrationHelpers on a
table that Postgres estimated to be non-empty when it was actually
empty.
2017-10-11 21:20:39 +02:00
Eugen Rochko 03457653e9 Bump version to 2.0.0rc2 2017-10-10 20:48:26 +02:00
Eugen Rochko 522fa878c3 Fix #5274 - Create symlink from public/500.html to public/assets/500.html (#5288) 2017-10-09 20:51:24 +02:00
Eugen Rochko 2e9c614e77 Bump version to 2.0.0rc1 (#5209) 2017-10-08 22:03:44 +02:00
Eugen Rochko 007fce8c10 Set snowflake IDs for backdated statuses (#5260)
- Rename Mastodon::TimestampIds into Mastodon::Snowflake for clarity
- Skip for statuses coming from inbox, aka delivered in real-time
- Skip for statuses that claim to be from the future
2017-10-08 17:34:34 +02:00
Nolan Lawson 2b34471d9d Compress and combine emoji data (#5229) 2017-10-06 03:42:34 +02:00
Eugen Rochko 27fd33256c Clean up code style of Mastodon::TimestampId module (#5232)
* Clean up code style of Mastodon::TimestampId module

* Update brakeman config
2017-10-06 03:42:21 +02:00
aschmitz 5b2d855d86 Non-Serial ("Snowflake") IDs (#4801)
* Use non-serial IDs

This change makes a number of nontrivial tweaks to the data model in
Mastodon:

* All IDs are now 8 byte integers (rather than mixed 4- and 8-byte)
* IDs are now assigned as:
  * Top 6 bytes: millisecond-resolution time from epoch
  * Bottom 2 bytes: serial (within the millisecond) sequence number
  * See /lib/tasks/db.rake's `define_timestamp_id` for details, but
    note that the purpose of these changes is to make it difficult to
    determine the number of objects in a table from the ID of any
    object.
* The Redis sorted set used for the feed will have values used to look
  up toots, rather than scores. This is almost always the same as the
  existing behavior, except in the case of boosted toots. This change
  was made because Redis stores scores as double-precision floats,
  which cannot store the new ID format exactly. Note that this doesn't
  cause problems with sorting/pagination, because ZREVRANGEBYSCORE
  sorts lexicographically when scores are tied. (This will still cause
  sorting issues when the ID gains a new significant digit, but that's
  extraordinarily uncommon.)

Note a couple of tradeoffs have been made in this commit:

* lib/tasks/db.rake is used to enforce many/most column constraints,
  because this commit seems likely to take a while to bring upstream.
  Enforcing a post-migrate hook is an easier way to maintain the code
  in the interim.
* Boosted toots will appear in the timeline as many times as they have
  been boosted. This is a tradeoff due to the way the feed is saved in
  Redis at the moment, but will be handled by a future commit.

This would effectively close Mastodon's #1059, as it is a
snowflake-like system of generating IDs. However, given how involved
the changes were simply within Mastodon, it may have unexpected
interactions with some clients, if they store IDs as doubles
(or as 4-byte integers). This was a problem that Twitter ran into with
their "snowflake" transition, particularly in JavaScript clients that
treated IDs as JS integers, rather than strings. It therefore would be
useful to test these changes at least in the web interface and popular
clients before pushing them to all users.

* Fix JavaScript interface with long IDs

Somewhat predictably, the JS interface handled IDs as numbers, which in
JS are IEEE double-precision floats. This loses some precision when
working with numbers as large as those generated by the new ID scheme,
so we instead handle them here as strings. This is relatively simple,
and doesn't appear to have caused any problems, but should definitely
be tested more thoroughly than the built-in tests. Several days of use
appear to support this working properly.

BREAKING CHANGE:

The major(!) change here is that IDs are now returned as strings by the
REST endpoints, rather than as integers. In practice, relatively few
changes were required to make the existing JS UI work with this change,
but it will likely hit API clients pretty hard: it's an entirely
different type to consume. (The one API client I tested, Tusky, handles
this with no problems, however.)

Twitter ran into this issue when introducing Snowflake IDs, and decided
to instead introduce an `id_str` field in JSON responses. I have opted
to *not* do that, and instead force all IDs to 64-bit integers
represented by strings in one go. (I believe Twitter exacerbated their
problem by rolling out the changes three times: once for statuses, once
for DMs, and once for user IDs, as well as by leaving an integer ID
value in JSON. As they said, "If you’re using the `id` field with JSON
in a Javascript-related language, there is a very high likelihood that
the integers will be silently munged by Javascript interpreters. In most
cases, this will result in behavior such as being unable to load or
delete a specific direct message, because the ID you're sending to the
API is different than the actual identifier associated with the
message." [1]) However, given that this is a significant change for API
users, alternatives or a transition time may be appropriate.

1: https://blog.twitter.com/developer/en_us/a/2011/direct-messages-going-snowflake-on-sep-30-2011.html

* Restructure feed pushes/unpushes

This was necessary because the previous behavior used Redis zset scores
to identify statuses, but those are IEEE double-precision floats, so we
can't actually use them to identify all 64-bit IDs. However, it leaves
the code in a much better state for refactoring reblog handling /
coalescing.

Feed-management code has been consolidated in FeedManager, including:

* BatchedRemoveStatusService no longer directly manipulates feed zsets
* RemoveStatusService no longer directly manipulates feed zsets
* PrecomputeFeedService has moved its logic to FeedManager#populate_feed

(PrecomputeFeedService largely made lots of calls to FeedManager, but
didn't follow the normal adding-to-feed process.)

This has the effect of unifying all of the feed push/unpush logic in
FeedManager, making it much more tractable to update it in the future.

Due to some additional checks that must be made during, for example,
batch status removals, some Redis pipelining has been removed. It does
not appear that this should cause significantly increased load, but if
necessary, some optimizations are possible in batch cases. These were
omitted in the pursuit of simplicity, but a batch_push and batch_unpush
would be possible in the future.

Tests were added to verify that pushes happen under expected conditions,
and to verify reblog behavior (both on pushing and unpushing). In the
case of unpushing, this includes testing behavior that currently leads
to confusion such as Mastodon's #2817, but this codifies that the
behavior is currently expected.

* Rubocop fixes

I could swear I made these changes already, but I must have lost them
somewhere along the line.

* Address review comments

This addresses the first two comments from review of this feature:

https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336735
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336931

This adds an optional argument to FeedManager#key, the subtype of feed
key to generate. It also tests to ensure that FeedManager's settings are
such that reblogs won't be tracked forever.

* Hardcode IdToBigints migration columns

This addresses a comment during review:
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139337452

This means we'll need to make sure that all _id columns going forward
are bigints, but that should happen automatically in most cases.

* Additional fixes for stringified IDs in JSON

These should be the last two. These were identified using eslint to try
to identify any plain casts to JavaScript numbers. (Some such casts are
legitimate, but these were not.)

Adding the following to .eslintrc.yml will identify casts to numbers:

~~~
  no-restricted-syntax:
  - warn
  - selector: UnaryExpression[operator='+'] > :not(Literal)
    message: Avoid the use of unary +
  - selector: CallExpression[callee.name='Number']
    message: Casting with Number() may coerce string IDs to numbers
~~~

The remaining three casts appear legitimate: two casts to array indices,
one in a server to turn an environment variable into a number.

* Only implement timestamp IDs for Status IDs

Per discussion in #4801, this is only being merged in for Status IDs at
this point. We do this in a migration, as there is no longer use for
a post-migration hook. We keep the initialization of the timestamp_id
function as a Rake task, as it is also needed after db:schema:load (as
db/schema.rb doesn't store Postgres functions).

* Change internal streaming payloads to stringified IDs as well

This is equivalent to 591a9af356faf2d5c7e66e3ec715502796c875cd from
#5019, with an extra change for the addition to FeedManager#unpush.

* Ensure we have a status_id_seq sequence

Apparently this is not a given when specifying a custom ID function,
so now we ensure it gets created. This uses the generic version of this
function to more easily support adding additional tables with timestamp
IDs in the future, although it would be possible to cut this down to a
less generic version if necessary. It is only run during db:schema:load
or the relevant migration, so the overhead is extraordinarily minimal.

* Transition reblogs to new Redis format

This provides a one-way migration to transition old Redis reblog entries
into the new format, with a separate tracking entry for reblogs.

It is not invertible because doing so could (if timestamp IDs are used)
require a database query for each status in each users' feed, which is
likely to be a significant toll on major instances.

* Address review comments from @akihikodaki

No functional changes.

* Additional review changes

* Heredoc cleanup

* Run db:schema:load hooks for test in development

This matches the behavior in Rails'
ActiveRecord::Tasks::DatabaseTasks.each_current_configuration, which
would otherwise break `rake db:setup` in development.

It also moves some functionality out to a library, which will be a good
place to put additional related functionality in the near future.
2017-10-04 09:56:37 +02:00
aschmitz 04fde500e6 Make IdsToBigints (mostly!) non-blocking (#5088)
* Make IdsToBigints (mostly!) non-blocking

This pulls in GitLab's MigrationHelpers, which include code to make
column changes in ways that Postgres can do without locking. In general,
this involves creating a new column, adding an index and any foreign
keys as appropriate, adding a trigger to keep it populated alongside
the old column, and then progressively copying data over to the new
column, before removing the old column and replacing it with the new
one.

A few changes to GitLab's MigrationHelpers were necessary:

* Some changes were made to remove dependencies on other GitLab code.
* We explicitly wait for index creation before forging ahead on column
  replacements.
* We use different temporary column names, to avoid running into index
  name length limits.
* We rename the generated indices back to what they "should" be after
  replacing columns.
* We rename the generated foreign keys to use the new column names when
  we had to create them. (This allows the migration to be rolled back
  without incident.)

# Big Scary Warning

There are two things here that may trip up large instances:

1. The change for tables' "id" columns is not concurrent. In
   particular, the stream_entries table may be big, and does not
   concurrently migrate its id column. (On the other hand, x_id type
   columns are all concurrent.)
2. This migration will take a long time to run, *but it should not
   lock tables during that time* (with the exception of the "id"
   columns as described above). That means this should probably be run
   in `screen` or some other session that can be run for a long time.
   Notably, the migration will take *longer* than it would without
   these changes, but the website will still be responsive during that
   time.

These changes were tested on a relatively large statuses table (256k
entries), and the service remained responsive during the migration.
Migrations both forward and backward were tested.

* Rubocop fixes

* MigrationHelpers: Support ID columns in some cases

This doesn't work in cases where the ID column is referred to as a
foreign key by another table.

* MigrationHelpers: support foreign keys for ID cols

Note that this does not yet support foreign keys on non-primary-key
columns, but Mastodon also doesn't yet have any that we've needed to
migrate.

This means we can perform fully "concurrent" migrations to change ID
column types, and the IdsToBigints migration can happen with effectively
no downtime. (A few operations require a transaction, such as renaming
columns or deleting them, but these transactions should not block for
noticeable amounts of time.)

The algorithm for generating foreign key names has changed with this,
and therefore all of those changed in schema.rb.

* Provide status, allow for interruptions

The MigrationHelpers now allow restarting the rename of a column if it
was interrupted, by removing the old "new column" and re-starting the
process.

Along with this, they now provide status updates on the changes which
are happening, as well as indications about when the changes can be
safely interrupted (when there are at least 10 seconds estimated to be
left before copying data is complete).

The IdsToBigints migration now also sorts the columns it migrates by
size, starting with the largest tables. This should provide
administrators a worst-case scenario estimate for the length of
migrations: each successive change will get faster, giving admins a
chance to abort early on if they need to run the migration later. The
idea is that this does not force them to try to time interruptions
between smaller migrations.

* Fix column sorting in IdsToBigints

Not a significant change, but it impacts the order of columns in the
database and db/schema.rb.

* Actually pause before IdsToBigints
2017-10-02 21:28:59 +02:00
Yamagishi Kazutoshi 12b7306a03 Upgrade Webpacker to version 3.0.1 (#5122) 2017-09-27 14:41:54 +02:00