* Fix uncaching worker
* Revert to using Paperclip's filesystem backend instead of fog-local
fog-local has lots of concurrency issues, causing failure to delete files,
dangling file records, and spurious errors UncacheMediaWorker
Conflicts:
app/javascript/mastodon/initial_state.js
db/schema.rb
Upstream added a new field to initial_state.
Not too sure about what happened with db/schema.rb though…
* No need to re-require sidekiq plugins, they are required via Gemfile
* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile
* Replace ruby-oembed with FetchOEmbedService
Reduce startup by 45382 allocated objects
* Remove preloaded JSON-LD in favour of caching HTTP responses
Reduce boot RAM by about 6 MiB
* Fix tests
* Fix test suite by stubbing out JSON-LD contexts
* Update annotate to version 2.7.3
* Update aws-sdk-s3 to version 1.9.2
* Update browser to version 2.5.3
* Update capistrano to version 3.10.2
* Update domain_name to version 0.5.20180417
* Update http to version 3.2.0
* Update lograge to version 0.10.0
* Update oj to version 3.5.1
* Update parallel_tests to version 2.21.3
* Update puma to version 3.11.4
* Update rubocop to version 0.55.0
* Update scss_lint to version 0.57.0
* Update simplecov to version 0.16.1
* Update tty-command to version 0.8.0
* Update tty-prompt to version 0.16.0
* Update pkg-config to version 1.3.0
* Update fog-local to version 0.5.0
* Update fog-openstack to version 0.1.25
* Update devise-two-factor to version 3.0.3
* bundle update
Conflicts:
app/serializers/initial_state_serializer.rb
The glitch flavour isn't yet pulling custom emoji data on its own (see
https://github.com/tootsuite/mastodon/pull/7047). Once that gets into
the glitch flavour, we can eliminate the custom_emojis load.
* update to new version of devise_pam_authenticatable2
* fix behaviour if suffix is nil, fix environment loading, fix user email creation
* code cleanup/fix linter warning
* Revert "Bump version to 2.3.2rc1"
This reverts commit cdf8b92fea.
* Revert "Downgrade Dockerfile to Ruby 2.4.3 on Alpine 3.6 (#6806)"
This reverts commit 0074cad44f.
* Revert "Handle Mastodon::HostValidationError when pulling remoteable assets (#6782)"
This reverts commit 4a0a19fe54.
* Revert "Correct the reference to user's password in mastodon:add_user task (#6800)"
This reverts commit 338bff8b93.
* Revert "Upgrade Paperclip to version 6.0.0 (#6754)"
This reverts commit b88fcd53f7.
* Fix#201: Account archive download
* Export actor and private key in the archive
* Optimize BackupService
- Add conversation to cached associations of status, because
somehow it was forgotten and is source of N+1 queries
- Explicitly call GC between batches of records being fetched
(Model class allocations are the worst offender)
- Stream media files into the tar in 1MB chunks
(Do not allocate media file (up to 8MB) as string into memory)
- Use #bytesize instead of #size to calculate file size for JSON
(Fix FileOverflow error)
- Segment media into subfolders by status ID because apparently
GIF-to-MP4 media are all named "media.mp4" for some reason
* Keep uniquely generated filename in Paperclip::GifTranscoder
* Ensure dumped files do not overwrite each other by maintaing directory partitions
* Give tar archives a good name
* Add scheduler to remove week-old backups
* Fix code style issue
* Fix avatar and header issues by using custom geometry detector
Revert a part of #6508. The file passed to dynamic styles method
was not actually a file, but an instance of Paperclip::Attachment,
which broke all styles by always returning {} from the method.
One problem with GIF avatars was that Paperclip::GeometryDetector
reported wrong dimensions for them, e.g. 120x120 GIF avatar would
for some reason be detected as 120x53. By writing our own geometry
parser, we can use FastImage, which also happens to be faster than
ImageMagick, to detect image dimensions, which are also correct.
Unfortunately, this PR does not implement skipping a `convert`
entirely if the dimensions are already correct, as I found no easy
way to write that behaviour into Paperclip without rewriting the
Paperclip::Thumbnail class.
* Only invoke convert if dimension or format needs to be changed
* Add better CLI prompt
* Add rake mastodon:setup interactive wizard
* Test db/redis/smtp configurations and add admin user at the end
* Test database connection even when database does not exist yet
Unlike strip_tags, html2text will preserve text present in other nodes,
e.g. anchor tags:
[1] pry(main)> str = '<a href="http://www.example.com">A link</a>'
=> "<a href=\"http://www.example.com\">A link</a>"
[2] pry(main)> Html2Text.convert(str)
=> "[A link](http://www.example.com)"
[3] pry(main)> include ActionView::Helpers::SanitizeHelper
=> Object
[4] pry(main)> strip_tags(str)
=> "A link"
Preserving the href of an anchor allows keyword mutes to also match on
URLs, which is something that the frontend regex filter can currently
do.
* Add full-text search for authorized statuses
- Search API will return statuses that match the query
- Only for logged in users
- Only if you are author of the status,
- Or you were mentioned in it
- Or you favourited or reblogged it
- Configuration over `ES_ENABLED`, `ES_HOST`, `ES_PORT`, `ES_PREFIX`
- Run `rails chewy:deploy` to create & populate index
Fix#5880Fix#4293Fix#1152
* Add commented out docker-compose configuration for ES container
* Optimize index import, filter search results
* Add basic normalization to the index
* Add better stemming and normalization to the index
* Skip webfinger request if search query includes both @ and a space
* Fix code style
* Visually separate search result sections
* Fix code style issues
* Add confirmation step for email changes
This adds a confirmation step for email changes of existing users.
Like the initial account confirmation, a confirmation link is sent
to the new address.
Additionally, a notification is sent to the existing address when
the change is initiated. This message includes instruction to reset
the password immediately or to contact the instance admin if the
change was not initiated by the account owner.
Fixes#3871
* Add review fixes
* Update active_model_serializers to version 0.10.7
* Update capistrano-rails to version 1.3.1
* Update capistrano-rbenv to version 2.1.3
* Update capybara to version 2.16.1
* Update devise-two-factor to version 3.0.2
* Update i18n-tasks to version 0.8.19
* Update ox to version 2.8.2
* Update parallel_tests to version 2.19.0
* Update puma to version 3.11.0
* Update redis-namespace to version 1.6.0
* Update rspec-rails to version 3.7.2
* Update scss_lint to version 0.56.0
* Update webmock to version 3.1.1
* Update webpush to version 0.3.3
* bundle update
* Update better_errors to version 2.4.0
* Update binding_of_caller to version 0.7.3
* Update bootsnap to version 1.1.5
* Update browser to version 2.5.2
* Update capistrano to version 3.10.0
* Update capistrano-bundler to version 1.3.0
* Update capistrano-rbenv to version 2.1.2
* Update capybara to version 2.15.4
* Update cld3 to version 3.2.1
* Update fabrication to version 2.18.0
* Update fog-openstack to version 0.1.22
* Update kaminari to version 1.1.1
* Update lograge to version 0.7.1
* Update nokogiri to version 1.8.1
* Update oj to version 3.3.9
* Update ox to version 2.8.1
* Update parallel_tests to version 2.17.0
* Update pkg-config to version 1.2.8
* Update rspec-rails to version 3.7.1
* Update rubocop to version 0.51.0
* Update scss_lint to version 0.55.0
* Update sidekiq to version 5.0.5
* Update sidekiq-scheduler to version 2.1.10
* Update tzinfo-data to version 1.2017.3
* Update webpacker to version 3.0.2
* bundle update
* Fix language filter codes
CLD3 returns BCP-47 language identifier, filter settings expect
identifiers in the ISO 639-1 format. Convert between formats,
and exclude duplicate languages from filter choices (zh-CN->zh)
* Fix zh name
* Add handling of Linked Data Signatures in payloads
* Add a way to sign JSON, fix canonicalization of signature options
* Fix signatureValue encoding, send out signed JSON when distributing
* Add missing security context
* Update twitter-text to version 1.14.7
* Update tilt to version 2.0.8
* Update statsd-instrument to version 2.1.4
* Update sidekiq to version 5.0.4
* Update sidekiq-scheduler to version 2.1.8
* Update sidekiq-unique-jobs to version 5.0.9
* Update redis-activesupport to version 5.0.3
* Update rails-settings-cached to version v0.6.6
* Update pkg-config to version 1.2.4
* Update parallel_tests to version 2.14.2
* Update jsonapi-renderer to version 0.1.3
* Update i18n-tasks to version 0.9.16
* Update httplog to version 0.99.7
* Update fabrication to version 2.16.2
* Update bootsnap to version 1.1.2
* Update aws-sigv4 to version 1.0.1
* Update aws-sdk-core to version 2.10.21
* Update hashdiff to version 0.3.5
* Update rails to version 5.1.3
* Refactor ResolveRemoteAccountService
* Remove trailing whitespace
* Use redis locks around critical ResolveRemoteAccountService code
* Add test for race condition of lock
* Add dependency on idn-ruby to speed up URI normalization
* Use normalized_host instead of normalize.host when applicable
When we are only interested in the normalized host, calling normalized_host
avoids normalizing the other components of the URI as well as creating a
new object
- Use unicode when selecting emoji through picker
- Convert shortcodes to unicode when storing text input server-side
- Do not convert shortcodes in JS anymore
* Specs for language detection
* Use CharlockHolmes instead of NKF
* Correct mistakes
* Correct style
* Set hint_enc instead of falling back and strip_tags
* Improve specs
* Add dependencies
* Update aws-sdk to version 2.10.4
* Update bootsnap to version 1.1.1
* Update capistrano to version 3.8.2
* Update capybara to version 2.14.4
* Update cld3 to version 3.1.3
* Update http_accept_language to version 2.1.1
* Update sidekiq to version 5.0.3
* Update rspec-sidekiq to version 3.0.3
* Update sidekiq-scheduler to version 2.1.7
* Update oj to version 3.2.0
* Update openssl to version 2.0.4
* Update pg to version 0.21.0
* Update twitter-text to version 1.14.6
* Update unicode-display_width to version 1.3.0
* Update scss_lint to version 0.54.0
* Update hamlit to version 2.8.4
* Update erubi to version 1.6.1
* Update httplog to version 0.99.4
* Update aws-sdk to version 2.10.6
* Add overview of active sessions
* Better display of browser/platform name
* Improve how browser information is stored and displayed for sessions overview
* Fix test
* Update mail to version 2.6.6
* Update aws-sdk to version 2.9.37
* Update capybara to version 2.14.2
* Update oj to version 3.1.0
* Update sidekiq to version 5.0.2
* Update puma to version 3.9.1
* Update sanitize to version 4.5.0
* Update capistrano-rails to version 1.3.0
* Update annotate to version 2.7.2
* Update puma to version 3.9.0
* Update aws-sdk to version 2.9.28
* Update bootsnap to version 1.0.0
* Update nio4r to version 2.1.0
* Update nokogumbo to version 1.4.12
* Update oj to version 3.0.11
* Update pkg-config to version 1.2.3
* Update rubocop to version 0.49.1
* Update sidekiq-scheduler to version 2.1.5
* Update rails to version 5.1.1
* Run `rails app:update`
* Remove the override of polymorphic activity relationship
* Silence warning about otp_secret attribute being unknown to rails
* We will only introduce form_with where we want to use remote data
* Update xpath to version 2.1.0
* Update aws-sdk to version 2.9.25
* Update oj to version 3.0.10
* Update brakeman to version 3.6.2
* Update rufus-scheduler to version 3.4.2
* Update doorkeeper to version 4.2.6
* Update rubocop to version 0.49.0
* Update terminal-table to version 1.8.0
* Update aws-sdk to version 2.9.20
* Update cld3 to version 3.1.2
* Update microformats2 to version 3.1.0
* Update rails-controller-testing to version 1.0.2
* Update oj to version 3.0.8
* Update http-form_data to version 1.0.3
* Update nokogumbo to version 1.4.11
* Update simplecov-html to version 0.10.1
* Update aws-sdk to version 2.9.21
* Update oj to version 3.0.9
* Update sass to version 3.4.24
* Update simple_form to version 3.5.0
* Update rails to version 5.0.3
* Update aws-sdk to version 2.9.17
* Update capistrano to version 3.8.1
* Update climate_control to version 0.2.0
* Update debug_inspector to version 0.0.3
* Update et-orbi to version 1.0.4
* Update oj to version 3.0.7
* Update microformats2 to version 2.9.0
* Update lograge to version 0.5.1
* Clean up usage of Microformats2 object to remove deprecation warnings
* Update microformats2 to version 3.0.1
Compact Language Detector v3 (CLD3) is the successor of CLD2, which was
used in the previous implementation. CLD3 includes improvements since CLD2,
and supports newer compilers. On the other hand, it has additional
requirements and cld3-ruby, the FFI of CLD3 for Ruby, is still new and may
be still inmature.
Though CLD3 is named after CLD2, it is implemented with a neural network
model, different from the old implementation, which is based on a Naïve
Bayesian classifier.
CLD3 supports newer compilers, such as GCC 6. CLD2 is not compatible with
GCC 6 because it assigns negative values to varibales typed unsigned.
(see internal/cld_generated_cjk_uni_prop_80.cc) The support for GCC 6 and
newer compilers are essential today, when some server operating system
such as Ubuntu Server 16.10 has GCC 6 by default.
On the one hand, CLD3 requires C++11 support. Environments with old
compilers such as Ubuntu Server 14.04 needs to update the system or install
a newer compiler.
CLD3 needs protocol buffers as a new dependency. However,it is not
considered problematic because major server operating systems, CentOS and
Ubuntu Server provide them.
The FFI cld3-ruby was written by me (Akihiko Odaki) for use in Mastodon.
It is still new and may be inmature, but confirmed to pass existing tests.
* Update bundler-audit to version 0.5.0
* Update oj to version 3.0.6
* Update http-form_data to version 1.0.2
* Update parallel to version 1.11.2
* Update rails-i18n to version 5.0.4
* Update rack-protection to version 2.0.0
* Update haml to version 5.0.1
* Update scss_lint to version 0.53.0
* Relax requirement on i18n-tasks
* Remove haml and haml_lint gem requirement
* add redis-namespace gem
* namespace redis usage
* refactor redis namespace code to be less intrusive
previously : would be prepended to keys when the
REDIS_NAMESPACE env var was not set
now if it is not set the namespacing functions are
not used at all, which should prevent disruptions
when instances update.
* fix redis namespace variable style in streaming js
* remove trailing space
* final redis namespace style fix
* Fix#2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron
Fix an issue where / in domain would raise exception in TagManager#normalize_domain
PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through
Fix#2702 - Correct user agent header on outgoing http requests
* Add test for SubscribeService
* Extract #expiring_accounts into method
* Make mastodon:push:refresh no-op
* Queues are now defined in sidekiq.yml
* Queues are now in sidekiq.yml
ThibG