* Fix uncaching worker
* Revert to using Paperclip's filesystem backend instead of fog-local
fog-local has lots of concurrency issues, causing failure to delete files,
dangling file records, and spurious errors UncacheMediaWorker
Conflicts:
app/javascript/mastodon/initial_state.js
db/schema.rb
Upstream added a new field to initial_state.
Not too sure about what happened with db/schema.rb though…
* No need to re-require sidekiq plugins, they are required via Gemfile
* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile
* Replace ruby-oembed with FetchOEmbedService
Reduce startup by 45382 allocated objects
* Remove preloaded JSON-LD in favour of caching HTTP responses
Reduce boot RAM by about 6 MiB
* Fix tests
* Fix test suite by stubbing out JSON-LD contexts
* Update annotate to version 2.7.3
* Update aws-sdk-s3 to version 1.9.2
* Update browser to version 2.5.3
* Update capistrano to version 3.10.2
* Update domain_name to version 0.5.20180417
* Update http to version 3.2.0
* Update lograge to version 0.10.0
* Update oj to version 3.5.1
* Update parallel_tests to version 2.21.3
* Update puma to version 3.11.4
* Update rubocop to version 0.55.0
* Update scss_lint to version 0.57.0
* Update simplecov to version 0.16.1
* Update tty-command to version 0.8.0
* Update tty-prompt to version 0.16.0
* Update pkg-config to version 1.3.0
* Update fog-local to version 0.5.0
* Update fog-openstack to version 0.1.25
* Update devise-two-factor to version 3.0.3
* bundle update
Conflicts:
app/serializers/initial_state_serializer.rb
The glitch flavour isn't yet pulling custom emoji data on its own (see
https://github.com/tootsuite/mastodon/pull/7047). Once that gets into
the glitch flavour, we can eliminate the custom_emojis load.
* update to new version of devise_pam_authenticatable2
* fix behaviour if suffix is nil, fix environment loading, fix user email creation
* code cleanup/fix linter warning
* Revert "Bump version to 2.3.2rc1"
This reverts commit cdf8b92fea.
* Revert "Downgrade Dockerfile to Ruby 2.4.3 on Alpine 3.6 (#6806)"
This reverts commit 0074cad44f.
* Revert "Handle Mastodon::HostValidationError when pulling remoteable assets (#6782)"
This reverts commit 4a0a19fe54.
* Revert "Correct the reference to user's password in mastodon:add_user task (#6800)"
This reverts commit 338bff8b93.
* Revert "Upgrade Paperclip to version 6.0.0 (#6754)"
This reverts commit b88fcd53f7.
* Fix#201: Account archive download
* Export actor and private key in the archive
* Optimize BackupService
- Add conversation to cached associations of status, because
somehow it was forgotten and is source of N+1 queries
- Explicitly call GC between batches of records being fetched
(Model class allocations are the worst offender)
- Stream media files into the tar in 1MB chunks
(Do not allocate media file (up to 8MB) as string into memory)
- Use #bytesize instead of #size to calculate file size for JSON
(Fix FileOverflow error)
- Segment media into subfolders by status ID because apparently
GIF-to-MP4 media are all named "media.mp4" for some reason
* Keep uniquely generated filename in Paperclip::GifTranscoder
* Ensure dumped files do not overwrite each other by maintaing directory partitions
* Give tar archives a good name
* Add scheduler to remove week-old backups
* Fix code style issue
* Fix avatar and header issues by using custom geometry detector
Revert a part of #6508. The file passed to dynamic styles method
was not actually a file, but an instance of Paperclip::Attachment,
which broke all styles by always returning {} from the method.
One problem with GIF avatars was that Paperclip::GeometryDetector
reported wrong dimensions for them, e.g. 120x120 GIF avatar would
for some reason be detected as 120x53. By writing our own geometry
parser, we can use FastImage, which also happens to be faster than
ImageMagick, to detect image dimensions, which are also correct.
Unfortunately, this PR does not implement skipping a `convert`
entirely if the dimensions are already correct, as I found no easy
way to write that behaviour into Paperclip without rewriting the
Paperclip::Thumbnail class.
* Only invoke convert if dimension or format needs to be changed
* Add better CLI prompt
* Add rake mastodon:setup interactive wizard
* Test db/redis/smtp configurations and add admin user at the end
* Test database connection even when database does not exist yet
Unlike strip_tags, html2text will preserve text present in other nodes,
e.g. anchor tags:
[1] pry(main)> str = '<a href="http://www.example.com">A link</a>'
=> "<a href=\"http://www.example.com\">A link</a>"
[2] pry(main)> Html2Text.convert(str)
=> "[A link](http://www.example.com)"
[3] pry(main)> include ActionView::Helpers::SanitizeHelper
=> Object
[4] pry(main)> strip_tags(str)
=> "A link"
Preserving the href of an anchor allows keyword mutes to also match on
URLs, which is something that the frontend regex filter can currently
do.
* Add full-text search for authorized statuses
- Search API will return statuses that match the query
- Only for logged in users
- Only if you are author of the status,
- Or you were mentioned in it
- Or you favourited or reblogged it
- Configuration over `ES_ENABLED`, `ES_HOST`, `ES_PORT`, `ES_PREFIX`
- Run `rails chewy:deploy` to create & populate index
Fix#5880Fix#4293Fix#1152
* Add commented out docker-compose configuration for ES container
* Optimize index import, filter search results
* Add basic normalization to the index
* Add better stemming and normalization to the index
* Skip webfinger request if search query includes both @ and a space
* Fix code style
* Visually separate search result sections
* Fix code style issues
* Add confirmation step for email changes
This adds a confirmation step for email changes of existing users.
Like the initial account confirmation, a confirmation link is sent
to the new address.
Additionally, a notification is sent to the existing address when
the change is initiated. This message includes instruction to reset
the password immediately or to contact the instance admin if the
change was not initiated by the account owner.
Fixes#3871
* Add review fixes
* Update active_model_serializers to version 0.10.7
* Update capistrano-rails to version 1.3.1
* Update capistrano-rbenv to version 2.1.3
* Update capybara to version 2.16.1
* Update devise-two-factor to version 3.0.2
* Update i18n-tasks to version 0.8.19
* Update ox to version 2.8.2
* Update parallel_tests to version 2.19.0
* Update puma to version 3.11.0
* Update redis-namespace to version 1.6.0
* Update rspec-rails to version 3.7.2
* Update scss_lint to version 0.56.0
* Update webmock to version 3.1.1
* Update webpush to version 0.3.3
* bundle update
* Update better_errors to version 2.4.0
* Update binding_of_caller to version 0.7.3
* Update bootsnap to version 1.1.5
* Update browser to version 2.5.2
* Update capistrano to version 3.10.0
* Update capistrano-bundler to version 1.3.0
* Update capistrano-rbenv to version 2.1.2
* Update capybara to version 2.15.4
* Update cld3 to version 3.2.1
* Update fabrication to version 2.18.0
* Update fog-openstack to version 0.1.22
* Update kaminari to version 1.1.1
* Update lograge to version 0.7.1
* Update nokogiri to version 1.8.1
* Update oj to version 3.3.9
* Update ox to version 2.8.1
* Update parallel_tests to version 2.17.0
* Update pkg-config to version 1.2.8
* Update rspec-rails to version 3.7.1
* Update rubocop to version 0.51.0
* Update scss_lint to version 0.55.0
* Update sidekiq to version 5.0.5
* Update sidekiq-scheduler to version 2.1.10
* Update tzinfo-data to version 1.2017.3
* Update webpacker to version 3.0.2
* bundle update
* Fix language filter codes
CLD3 returns BCP-47 language identifier, filter settings expect
identifiers in the ISO 639-1 format. Convert between formats,
and exclude duplicate languages from filter choices (zh-CN->zh)
* Fix zh name
* Add handling of Linked Data Signatures in payloads
* Add a way to sign JSON, fix canonicalization of signature options
* Fix signatureValue encoding, send out signed JSON when distributing
* Add missing security context
* Update twitter-text to version 1.14.7
* Update tilt to version 2.0.8
* Update statsd-instrument to version 2.1.4
* Update sidekiq to version 5.0.4
* Update sidekiq-scheduler to version 2.1.8
* Update sidekiq-unique-jobs to version 5.0.9
* Update redis-activesupport to version 5.0.3
* Update rails-settings-cached to version v0.6.6
* Update pkg-config to version 1.2.4
* Update parallel_tests to version 2.14.2
* Update jsonapi-renderer to version 0.1.3
* Update i18n-tasks to version 0.9.16
* Update httplog to version 0.99.7
* Update fabrication to version 2.16.2
* Update bootsnap to version 1.1.2
* Update aws-sigv4 to version 1.0.1
* Update aws-sdk-core to version 2.10.21
* Update hashdiff to version 0.3.5
* Update rails to version 5.1.3
* Refactor ResolveRemoteAccountService
* Remove trailing whitespace
* Use redis locks around critical ResolveRemoteAccountService code
* Add test for race condition of lock
* Add dependency on idn-ruby to speed up URI normalization
* Use normalized_host instead of normalize.host when applicable
When we are only interested in the normalized host, calling normalized_host
avoids normalizing the other components of the URI as well as creating a
new object
- Use unicode when selecting emoji through picker
- Convert shortcodes to unicode when storing text input server-side
- Do not convert shortcodes in JS anymore
* Specs for language detection
* Use CharlockHolmes instead of NKF
* Correct mistakes
* Correct style
* Set hint_enc instead of falling back and strip_tags
* Improve specs
* Add dependencies
* Update aws-sdk to version 2.10.4
* Update bootsnap to version 1.1.1
* Update capistrano to version 3.8.2
* Update capybara to version 2.14.4
* Update cld3 to version 3.1.3
* Update http_accept_language to version 2.1.1
* Update sidekiq to version 5.0.3
* Update rspec-sidekiq to version 3.0.3
* Update sidekiq-scheduler to version 2.1.7
* Update oj to version 3.2.0
* Update openssl to version 2.0.4
* Update pg to version 0.21.0
* Update twitter-text to version 1.14.6
* Update unicode-display_width to version 1.3.0
* Update scss_lint to version 0.54.0
* Update hamlit to version 2.8.4
* Update erubi to version 1.6.1
* Update httplog to version 0.99.4
* Update aws-sdk to version 2.10.6
* Add overview of active sessions
* Better display of browser/platform name
* Improve how browser information is stored and displayed for sessions overview
* Fix test
* Update mail to version 2.6.6
* Update aws-sdk to version 2.9.37
* Update capybara to version 2.14.2
* Update oj to version 3.1.0
* Update sidekiq to version 5.0.2
* Update puma to version 3.9.1
* Update sanitize to version 4.5.0
* Update capistrano-rails to version 1.3.0
* Update annotate to version 2.7.2
* Update puma to version 3.9.0
* Update aws-sdk to version 2.9.28
* Update bootsnap to version 1.0.0
* Update nio4r to version 2.1.0
* Update nokogumbo to version 1.4.12
* Update oj to version 3.0.11
* Update pkg-config to version 1.2.3
* Update rubocop to version 0.49.1
* Update sidekiq-scheduler to version 2.1.5
* Update rails to version 5.1.1
* Run `rails app:update`
* Remove the override of polymorphic activity relationship
* Silence warning about otp_secret attribute being unknown to rails
* We will only introduce form_with where we want to use remote data
* Update xpath to version 2.1.0
* Update aws-sdk to version 2.9.25
* Update oj to version 3.0.10
* Update brakeman to version 3.6.2
* Update rufus-scheduler to version 3.4.2
* Update doorkeeper to version 4.2.6
* Update rubocop to version 0.49.0
* Update terminal-table to version 1.8.0
* Update aws-sdk to version 2.9.20
* Update cld3 to version 3.1.2
* Update microformats2 to version 3.1.0
* Update rails-controller-testing to version 1.0.2
* Update oj to version 3.0.8
* Update http-form_data to version 1.0.3
* Update nokogumbo to version 1.4.11
* Update simplecov-html to version 0.10.1
* Update aws-sdk to version 2.9.21
* Update oj to version 3.0.9
* Update sass to version 3.4.24
* Update simple_form to version 3.5.0
* Update rails to version 5.0.3
* Update aws-sdk to version 2.9.17
* Update capistrano to version 3.8.1
* Update climate_control to version 0.2.0
* Update debug_inspector to version 0.0.3
* Update et-orbi to version 1.0.4
* Update oj to version 3.0.7
* Update microformats2 to version 2.9.0
* Update lograge to version 0.5.1
* Clean up usage of Microformats2 object to remove deprecation warnings
* Update microformats2 to version 3.0.1
Compact Language Detector v3 (CLD3) is the successor of CLD2, which was
used in the previous implementation. CLD3 includes improvements since CLD2,
and supports newer compilers. On the other hand, it has additional
requirements and cld3-ruby, the FFI of CLD3 for Ruby, is still new and may
be still inmature.
Though CLD3 is named after CLD2, it is implemented with a neural network
model, different from the old implementation, which is based on a Naïve
Bayesian classifier.
CLD3 supports newer compilers, such as GCC 6. CLD2 is not compatible with
GCC 6 because it assigns negative values to varibales typed unsigned.
(see internal/cld_generated_cjk_uni_prop_80.cc) The support for GCC 6 and
newer compilers are essential today, when some server operating system
such as Ubuntu Server 16.10 has GCC 6 by default.
On the one hand, CLD3 requires C++11 support. Environments with old
compilers such as Ubuntu Server 14.04 needs to update the system or install
a newer compiler.
CLD3 needs protocol buffers as a new dependency. However,it is not
considered problematic because major server operating systems, CentOS and
Ubuntu Server provide them.
The FFI cld3-ruby was written by me (Akihiko Odaki) for use in Mastodon.
It is still new and may be inmature, but confirmed to pass existing tests.
* Update bundler-audit to version 0.5.0
* Update oj to version 3.0.6
* Update http-form_data to version 1.0.2
* Update parallel to version 1.11.2
* Update rails-i18n to version 5.0.4
* Update rack-protection to version 2.0.0
* Update haml to version 5.0.1
* Update scss_lint to version 0.53.0
* Relax requirement on i18n-tasks
* Remove haml and haml_lint gem requirement
* add redis-namespace gem
* namespace redis usage
* refactor redis namespace code to be less intrusive
previously : would be prepended to keys when the
REDIS_NAMESPACE env var was not set
now if it is not set the namespacing functions are
not used at all, which should prevent disruptions
when instances update.
* fix redis namespace variable style in streaming js
* remove trailing space
* final redis namespace style fix
* Fix#2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron
Fix an issue where / in domain would raise exception in TagManager#normalize_domain
PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through
Fix#2702 - Correct user agent header on outgoing http requests
* Add test for SubscribeService
* Extract #expiring_accounts into method
* Make mastodon:push:refresh no-op
* Queues are now defined in sidekiq.yml
* Queues are now in sidekiq.yml
* Replace browserify with webpack
* Add react-intl-translations-manager
* Do not minify in development, add offline-plugin for ServiceWorker background cache updates
* Adjust tests and dependencies
* Fix production deployments
* Fix tests
* More optimizations
* Improve travis cache for npm stuff
* Re-run travis
* Add back support for custom.scss as before
* Remove offline-plugin and babili
* Fix issue with Immutable.List().unshift(...values) not working as expected
* Make travis load schema instead of running all migrations in sequence
* Fix missing React import in WarningContainer. Optimize rendering performance by using ImmutablePureComponent instead of
React.PureComponent. ImmutablePureComponent uses Immutable.is() to compare props. Replace dynamic callback bindings in
<UI />
* Add react definitions to places that use JSX
* Add Procfile.dev for running rails, webpack and streaming API at the same time
* add annotate to Gemfile
* rails g annotate:install
* configure annotate_models
* add schema info to models
* fix rubocop to add frozen_string_literal
* Update autoprefixer-rails to version 6.7.7.2
* Update aws-sdk to version 2.9.12
* Update http to version 2.2.2
* Update mail to version 2.6.5
* Update pghero to version 1.6.5
* Update ox to version 2.4.13
* Update json to version 2.1.0
* Update capistrano-rbenv to version 2.1.1
* Update httplog to version 0.99.3
* Update rainbow to version 2.2.2
* Update pkg-config to version 1.2.0
* Update globalid to version 0.4.0
* Update redis-rack to version 2.0.2
* Update webmock to version 3.0.1
* Update oj to version 3.0.2
* Instead of parsing shared status contents verbatim, make roundtrip
to purported original URL. Confirm that the "original" URL is from the
same domain as the author it claims to be from.
* Fix obvious typo, add comment
* Use URI look-up first
* Add test, update Goldfinger dependency to make less useless HTTP requests per Webfinger lookup
* OEmbed support for PreviewCard
* Improve ProviderDiscovery code failure treatment
* Do not crawl links if there is a content warning, since those
don't display a link card anyway
* Reset db schema
* Fresh migrate
* Fix rubocop style issues
Fix#1681 - return existing access token when applicable instead of creating new
* Fix test
* Extract http client to helper
* Improve oembed controller
For the 'time ago' value of stream entry simple statuses and the timestamp value
of detailed statuses - output their UTC value and use the browser timezone to
convert locally.
* Add language detection via WhatLanguage and (de)serialization of it through Atom
* Fix default language in ProcessFeedService
* Re-add newline before 'react-rails' Gem to fix groupings
Fixes Code Climate issue
This commit introduces Capybara and the first feature spec.
I focused on coverage for log in for the first feature spec because that would
have prevented 624a9a7136 causing #1236.
The pkg-config gem was necessary in order for nokogiri to find the system
libraries when building on OpenBSD.
Closes#1637
Signed-off-by: Bryce Chidester <bryce@cobryce.com>
* Replace will_paginate with kaminari
* Use #page instead of #paginate in controllers
* Replace will_paginate.page_gap with pagination.truncate in i18n
* Customize kaminari views to match prior styles
* Set kaminari options to match prior behavior
* Replace will_paginate with paginate in views
* Update rspec-rails to version 3.5.2
* Update addressable to version 2.5.1
* Update autoprefixer-rails to version 6.7.7.1
* Update bullet to version 5.5.1
* Update domain_name to version 0.5.20170404
* Update letter_opener_web to version 1.3.1
* Upate redis-rails to version 5.0.2
* Update active_record_query_trace to version 1.5.4
* Update capistrano-rails to version 1.2.3
* Update dotenv-rails to version 2.2.0
* Update pg to version 0.20.0
* Update tilt to version 2.0.7
* Update warden to version 1.2.7
* Update tins to version 1.13.2
* Update terminal-table to version 1.7.3
* Update oj to version 2.18.5
* Update simplecov to version 0.14.1
* Update uglifier to version 3.1.13
* Update hashdiff to version 0.3.2
* Update webmock to version 2.3.2
* Update devise to version 4.2.1
* Use ruby version 2.4.1
* Update sass to version 3.4.23
* Update puma to version 3.8.2
* Update will_paginate to version 3.1.5
* Update font-awesome-rails to version 4.7.0.1
* Update fuubar to version 2.2.0
* Update pry-rails to version 0.3.6
* Update simple-navigation to version 4.0.5
* Update rubocop to version 0.48.1
* Update doorkeeper to version 4.2.5
* Update faker to version 1.7.3
* Update aws-sdk to version 2.9.5
* Update fabrication to version 2.16.1
* Update hamlit-rails to version 0.2.0
* Update http to version 2.2.1
* Update httplog to version 0.99.2
* Update sidekiq to version 4.2.10
* Update rspec-sidekiq to version 3.0.0
* Update pghero to version 1.6.4
* Update rack-cors to version 0.4.1
* Update i18n-tasks to version 0.9.13
* Update ruby-oembed to version 0.12.0
* Update jquery-rails to version 4.3.1
* Update simple_form to version 3.4.0
* Update react-rails to version 1.11.0
* Update aws-sdk to version 2.9.6
* Update sidekiq-unique-jobs to version 5.0.0
* Update uglifier to version 3.2.0
* Use HTTP Accept-Language to detect locale
* Fix gem order to comply with codeclimate
* Sort gem to comply with rubocop
* I18n.default_locale fallback when there is no accept-language header
* Rewrite Atom generation from stream entries to use Ox instead of Nokogiri::Builder
StreamEntry is now limited to only statuses, which allows some optimization. Removed
extra queries on AccountsController#show. AtomSerializer instead of AtomBuilderHelper
used in AccountsController#show, StreamEntriesController#show, StreamEntryRenderer
and PubSubHubbub::DistributionWorker
PubSubHubbub::DistributionWorker moves n+1 DomainBlock query to PubSubHubbub::DeliveryWorker
instead.
All Salmon slaps that aren't based on StreamEntry still use AtomBuilderHelper and Nokogiri
* All Salmon slaps now use Ox instead of Nokogiri. No touch from status on account
This is a big one, so let me enumerate:
Accounts as well as stream entry pages now contain Link headers that
reference the Atom feed and Webfinger URL for the former and Atom entry
for the latter. So you only need to HEAD those resources to get that
information, no need to download and parse HTML <link>s.
ProcessFeedService will now queue ThreadResolveWorker for each remote
status that it cannot find otherwise. Furthermore, entries are now
processed in reverse order (from bottom to top) in case a newer entry
references a chronologically previous one.
ThreadResolveWorker uses FetchRemoteStatusService to obtain a status
and attach the child status it was queued for to it.
FetchRemoteStatusService looks up the URL, first with a HEAD, tests
if it's an Atom feed, in which case it processes it directly. Next
for Link headers to the Atom feed, in which case that is fetched
and processed. Lastly if it's HTML, it is checked for <link>s to the Atom
feed, and if such is found, that is fetched and processed. The account for
the status is derived from author/name attribute in the XML and the hostname
in the URL (domain). FollowRemoteAccountService and ProcessFeedService
are used.
This means that potentially threads are resolved recursively until a dead-end
is encountered, however it is performed asynchronously over background jobs,
so it should be ok.
POST /api/statuses Params: status (text contents), in_reply_to_id (optional)
GET /api/statuses/:id
POST /api/statuses/:id/reblog
GET /api/accounts/:id
GET /api/accounts/:id/following
GET /api/accounts/:id/followers
POST /api/accounts/:id/follow
POST /api/accounts/:id/unfollow
POST /api/follows Params: uri (e.g. user@domain)
OAuth authentication is currently disabled, but the API can be used with HTTP Auth.
ThibG