Compact Language Detector v3 (CLD3) is the successor of CLD2, which was
used in the previous implementation. CLD3 includes improvements since CLD2,
and supports newer compilers. On the other hand, it has additional
requirements and cld3-ruby, the FFI of CLD3 for Ruby, is still new and may
be still inmature.
Though CLD3 is named after CLD2, it is implemented with a neural network
model, different from the old implementation, which is based on a Naïve
Bayesian classifier.
CLD3 supports newer compilers, such as GCC 6. CLD2 is not compatible with
GCC 6 because it assigns negative values to varibales typed unsigned.
(see internal/cld_generated_cjk_uni_prop_80.cc) The support for GCC 6 and
newer compilers are essential today, when some server operating system
such as Ubuntu Server 16.10 has GCC 6 by default.
On the one hand, CLD3 requires C++11 support. Environments with old
compilers such as Ubuntu Server 14.04 needs to update the system or install
a newer compiler.
CLD3 needs protocol buffers as a new dependency. However,it is not
considered problematic because major server operating systems, CentOS and
Ubuntu Server provide them.
The FFI cld3-ruby was written by me (Akihiko Odaki) for use in Mastodon.
It is still new and may be inmature, but confirmed to pass existing tests.
* Update bundler-audit to version 0.5.0
* Update oj to version 3.0.6
* Update http-form_data to version 1.0.2
* Update parallel to version 1.11.2
* Update rails-i18n to version 5.0.4
* Update rack-protection to version 2.0.0
* Update haml to version 5.0.1
* Update scss_lint to version 0.53.0
* Relax requirement on i18n-tasks
* Remove haml and haml_lint gem requirement
* add redis-namespace gem
* namespace redis usage
* refactor redis namespace code to be less intrusive
previously : would be prepended to keys when the
REDIS_NAMESPACE env var was not set
now if it is not set the namespacing functions are
not used at all, which should prevent disruptions
when instances update.
* fix redis namespace variable style in streaming js
* remove trailing space
* final redis namespace style fix
* Fix#2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron
Fix an issue where / in domain would raise exception in TagManager#normalize_domain
PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through
Fix#2702 - Correct user agent header on outgoing http requests
* Add test for SubscribeService
* Extract #expiring_accounts into method
* Make mastodon:push:refresh no-op
* Queues are now defined in sidekiq.yml
* Queues are now in sidekiq.yml
* Replace browserify with webpack
* Add react-intl-translations-manager
* Do not minify in development, add offline-plugin for ServiceWorker background cache updates
* Adjust tests and dependencies
* Fix production deployments
* Fix tests
* More optimizations
* Improve travis cache for npm stuff
* Re-run travis
* Add back support for custom.scss as before
* Remove offline-plugin and babili
* Fix issue with Immutable.List().unshift(...values) not working as expected
* Make travis load schema instead of running all migrations in sequence
* Fix missing React import in WarningContainer. Optimize rendering performance by using ImmutablePureComponent instead of
React.PureComponent. ImmutablePureComponent uses Immutable.is() to compare props. Replace dynamic callback bindings in
<UI />
* Add react definitions to places that use JSX
* Add Procfile.dev for running rails, webpack and streaming API at the same time
* add annotate to Gemfile
* rails g annotate:install
* configure annotate_models
* add schema info to models
* fix rubocop to add frozen_string_literal
* Update autoprefixer-rails to version 6.7.7.2
* Update aws-sdk to version 2.9.12
* Update http to version 2.2.2
* Update mail to version 2.6.5
* Update pghero to version 1.6.5
* Update ox to version 2.4.13
* Update json to version 2.1.0
* Update capistrano-rbenv to version 2.1.1
* Update httplog to version 0.99.3
* Update rainbow to version 2.2.2
* Update pkg-config to version 1.2.0
* Update globalid to version 0.4.0
* Update redis-rack to version 2.0.2
* Update webmock to version 3.0.1
* Update oj to version 3.0.2
* Instead of parsing shared status contents verbatim, make roundtrip
to purported original URL. Confirm that the "original" URL is from the
same domain as the author it claims to be from.
* Fix obvious typo, add comment
* Use URI look-up first
* Add test, update Goldfinger dependency to make less useless HTTP requests per Webfinger lookup
* OEmbed support for PreviewCard
* Improve ProviderDiscovery code failure treatment
* Do not crawl links if there is a content warning, since those
don't display a link card anyway
* Reset db schema
* Fresh migrate
* Fix rubocop style issues
Fix#1681 - return existing access token when applicable instead of creating new
* Fix test
* Extract http client to helper
* Improve oembed controller
For the 'time ago' value of stream entry simple statuses and the timestamp value
of detailed statuses - output their UTC value and use the browser timezone to
convert locally.
* Add language detection via WhatLanguage and (de)serialization of it through Atom
* Fix default language in ProcessFeedService
* Re-add newline before 'react-rails' Gem to fix groupings
Fixes Code Climate issue
This commit introduces Capybara and the first feature spec.
I focused on coverage for log in for the first feature spec because that would
have prevented a1c63cb01d causing #1236.
The pkg-config gem was necessary in order for nokogiri to find the system
libraries when building on OpenBSD.
Closes#1637
Signed-off-by: Bryce Chidester <bryce@cobryce.com>
* Replace will_paginate with kaminari
* Use #page instead of #paginate in controllers
* Replace will_paginate.page_gap with pagination.truncate in i18n
* Customize kaminari views to match prior styles
* Set kaminari options to match prior behavior
* Replace will_paginate with paginate in views
* Update rspec-rails to version 3.5.2
* Update addressable to version 2.5.1
* Update autoprefixer-rails to version 6.7.7.1
* Update bullet to version 5.5.1
* Update domain_name to version 0.5.20170404
* Update letter_opener_web to version 1.3.1
* Upate redis-rails to version 5.0.2
* Update active_record_query_trace to version 1.5.4
* Update capistrano-rails to version 1.2.3
* Update dotenv-rails to version 2.2.0
* Update pg to version 0.20.0
* Update tilt to version 2.0.7
* Update warden to version 1.2.7
* Update tins to version 1.13.2
* Update terminal-table to version 1.7.3
* Update oj to version 2.18.5
* Update simplecov to version 0.14.1
* Update uglifier to version 3.1.13
* Update hashdiff to version 0.3.2
* Update webmock to version 2.3.2
* Update devise to version 4.2.1
* Use ruby version 2.4.1
* Update sass to version 3.4.23
* Update puma to version 3.8.2
* Update will_paginate to version 3.1.5
* Update font-awesome-rails to version 4.7.0.1
* Update fuubar to version 2.2.0
* Update pry-rails to version 0.3.6
* Update simple-navigation to version 4.0.5
* Update rubocop to version 0.48.1
* Update doorkeeper to version 4.2.5
* Update faker to version 1.7.3
* Update aws-sdk to version 2.9.5
* Update fabrication to version 2.16.1
* Update hamlit-rails to version 0.2.0
* Update http to version 2.2.1
* Update httplog to version 0.99.2
* Update sidekiq to version 4.2.10
* Update rspec-sidekiq to version 3.0.0
* Update pghero to version 1.6.4
* Update rack-cors to version 0.4.1
* Update i18n-tasks to version 0.9.13
* Update ruby-oembed to version 0.12.0
* Update jquery-rails to version 4.3.1
* Update simple_form to version 3.4.0
* Update react-rails to version 1.11.0
* Update aws-sdk to version 2.9.6
* Update sidekiq-unique-jobs to version 5.0.0
* Update uglifier to version 3.2.0
* Use HTTP Accept-Language to detect locale
* Fix gem order to comply with codeclimate
* Sort gem to comply with rubocop
* I18n.default_locale fallback when there is no accept-language header
* Rewrite Atom generation from stream entries to use Ox instead of Nokogiri::Builder
StreamEntry is now limited to only statuses, which allows some optimization. Removed
extra queries on AccountsController#show. AtomSerializer instead of AtomBuilderHelper
used in AccountsController#show, StreamEntriesController#show, StreamEntryRenderer
and PubSubHubbub::DistributionWorker
PubSubHubbub::DistributionWorker moves n+1 DomainBlock query to PubSubHubbub::DeliveryWorker
instead.
All Salmon slaps that aren't based on StreamEntry still use AtomBuilderHelper and Nokogiri
* All Salmon slaps now use Ox instead of Nokogiri. No touch from status on account
This is a big one, so let me enumerate:
Accounts as well as stream entry pages now contain Link headers that
reference the Atom feed and Webfinger URL for the former and Atom entry
for the latter. So you only need to HEAD those resources to get that
information, no need to download and parse HTML <link>s.
ProcessFeedService will now queue ThreadResolveWorker for each remote
status that it cannot find otherwise. Furthermore, entries are now
processed in reverse order (from bottom to top) in case a newer entry
references a chronologically previous one.
ThreadResolveWorker uses FetchRemoteStatusService to obtain a status
and attach the child status it was queued for to it.
FetchRemoteStatusService looks up the URL, first with a HEAD, tests
if it's an Atom feed, in which case it processes it directly. Next
for Link headers to the Atom feed, in which case that is fetched
and processed. Lastly if it's HTML, it is checked for <link>s to the Atom
feed, and if such is found, that is fetched and processed. The account for
the status is derived from author/name attribute in the XML and the hostname
in the URL (domain). FollowRemoteAccountService and ProcessFeedService
are used.
This means that potentially threads are resolved recursively until a dead-end
is encountered, however it is performed asynchronously over background jobs,
so it should be ok.
Jack Jennings