Compact Language Detector v3 (CLD3) is the successor of CLD2, which was
used in the previous implementation. CLD3 includes improvements since CLD2,
and supports newer compilers. On the other hand, it has additional
requirements and cld3-ruby, the FFI of CLD3 for Ruby, is still new and may
be still inmature.

Though CLD3 is named after CLD2, it is implemented with a neural network
model, different from the old implementation, which is based on a Naïve
Bayesian classifier.

CLD3 supports newer compilers, such as GCC 6. CLD2 is not compatible with
GCC 6 because it assigns negative values to varibales typed unsigned.
(see internal/cld_generated_cjk_uni_prop_80.cc) The support for GCC 6 and
newer compilers are essential today, when some server operating system
such as Ubuntu Server 16.10 has GCC 6 by default.

On the one hand, CLD3 requires C++11 support. Environments with old
compilers such as Ubuntu Server 14.04 needs to update the system or install
a newer compiler.

CLD3 needs protocol buffers as a new dependency. However,it is not
considered problematic because major server operating systems, CentOS and
Ubuntu Server provide them.

The FFI cld3-ruby was written by me (Akihiko Odaki) for use in Mastodon.
It is still new and may be inmature, but confirmed to pass existing tests.
rebase/4.0.0rc2
Akihiko Odaki 2017-05-10 02:58:03 +09:00 committed by Eugen Rochko
parent af6a84da14
commit d5cabfe5c6
7 changed files with 23 additions and 15 deletions

View File

@ -13,9 +13,9 @@ env:
- LOCAL_DOMAIN=cb6e6126.ngrok.io - LOCAL_DOMAIN=cb6e6126.ngrok.io
- LOCAL_HTTPS=true - LOCAL_HTTPS=true
- RAILS_ENV=test - RAILS_ENV=test
- CXX=g++-4.8
- NOKOGIRI_USE_SYSTEM_LIBRARIES=true - NOKOGIRI_USE_SYSTEM_LIBRARIES=true
- PARALLEL_TEST_PROCESSORS=2 - PARALLEL_TEST_PROCESSORS=2
- "PATH=$HOME:$PATH"
addons: addons:
postgresql: 9.4 postgresql: 9.4
@ -24,8 +24,10 @@ addons:
- ubuntu-toolchain-r-test - ubuntu-toolchain-r-test
- trusty-media - trusty-media
packages: packages:
- g++-4.8
- ffmpeg - ffmpeg
- g++-6
- libprotobuf-dev
- protobuf-compiler
rvm: rvm:
- 2.3.4 - 2.3.4
@ -43,6 +45,7 @@ install:
before_script: before_script:
- bundle exec rake parallel:create parallel:load_schema parallel:prepare - bundle exec rake parallel:create parallel:load_schema parallel:prepare
- bundle exec rails assets:precompile - bundle exec rails assets:precompile
- ln -s /usr/bin/x86_64-linux-gnu-g++-6 "$HOME/g++"
script: script:
- bundle exec parallel_test spec/ --group-by filesize --type rspec - bundle exec parallel_test spec/ --group-by filesize --type rspec

View File

@ -16,7 +16,8 @@ RUN echo "@edge https://nl.alpinelinux.org/alpine/edge/main" >> /etc/apk/reposit
libxml2-dev \ libxml2-dev \
libxslt-dev \ libxslt-dev \
python \ python \
build-base" \ build-base \
protobuf-dev" \
&& apk -U upgrade && apk add \ && apk -U upgrade && apk add \
$BUILD_DEPS \ $BUILD_DEPS \
nodejs@edge \ nodejs@edge \
@ -29,6 +30,7 @@ RUN echo "@edge https://nl.alpinelinux.org/alpine/edge/main" >> /etc/apk/reposit
file \ file \
imagemagick@edge \ imagemagick@edge \
ca-certificates \ ca-certificates \
protobuf \
&& npm install -g npm@3 && npm install -g yarn \ && npm install -g npm@3 && npm install -g yarn \
&& update-ca-certificates \ && update-ca-certificates \
&& rm -rf /tmp/* /var/cache/apk/* && rm -rf /tmp/* /var/cache/apk/*

View File

@ -19,7 +19,7 @@ gem 'paperclip', '~> 5.1'
gem 'paperclip-av-transcoder' gem 'paperclip-av-transcoder'
gem 'addressable' gem 'addressable'
gem 'cld2', require: 'cld' gem 'cld3', '~> 3.1.0'
gem 'devise' gem 'devise'
gem 'devise-two-factor' gem 'devise-two-factor'
gem 'doorkeeper' gem 'doorkeeper'

View File

@ -99,8 +99,8 @@ GEM
rack-test (>= 0.5.4) rack-test (>= 0.5.4)
xpath (~> 2.0) xpath (~> 2.0)
chunky_png (1.3.8) chunky_png (1.3.8)
cld2 (1.0.3) cld3 (3.1.0)
ffi (~> 1.9.3) ffi (>= 1.1.0, < 1.10.0)
climate_control (0.1.0) climate_control (0.1.0)
cocaine (0.5.8) cocaine (0.5.8)
climate_control (>= 0.0.3, < 1.0) climate_control (>= 0.0.3, < 1.0)
@ -480,7 +480,7 @@ DEPENDENCIES
capistrano-rbenv capistrano-rbenv
capistrano-yarn capistrano-yarn
capybara capybara
cld2 cld3 (~> 3.1.0)
devise devise
devise-two-factor devise-two-factor
doorkeeper doorkeeper

2
Vagrantfile vendored
View File

@ -33,7 +33,9 @@ sudo apt-get install \
redis-tools \ redis-tools \
postgresql \ postgresql \
postgresql-contrib \ postgresql-contrib \
protobuf-compiler \
yarn \ yarn \
libprotobuf-dev \
libreadline-dev \ libreadline-dev \
-y -y

View File

@ -6,6 +6,7 @@ class LanguageDetector
def initialize(text, account = nil) def initialize(text, account = nil)
@text = text @text = text
@account = account @account = account
@identifier = CLD3::NNetLanguageIdentifier.new(1, 2048)
end end
def to_iso_s def to_iso_s
@ -15,15 +16,15 @@ class LanguageDetector
private private
def detected_language_code def detected_language_code
detected_language[:code].to_sym if detected_language_reliable? result.language.to_sym if detected_language_reliable?
end end
def detected_language def result
@_detected_language ||= CLD.detect_language(text_without_urls) @result ||= @identifier.find_language(text_without_urls)
end end
def detected_language_reliable? def detected_language_reliable?
detected_language[:reliable] result.reliable?
end end
def text_without_urls def text_without_urls

View File

@ -24,15 +24,15 @@ describe LanguageDetector do
end end
describe 'when language can\'t be detected' do describe 'when language can\'t be detected' do
it 'confirm language engine cant detect' do it 'uses default locale when sent an empty document' do
result = CLD.detect_language('') result = described_class.new('').to_iso_s
expect(result[:reliable]).to be false expect(result).to eq :en
end end
describe 'because of a URL' do describe 'because of a URL' do
it 'uses default locale when sent just a URL' do it 'uses default locale when sent just a URL' do
string = 'http://example.com/media/2kFTgOJLXhQf0g2nKB4' string = 'http://example.com/media/2kFTgOJLXhQf0g2nKB4'
cld_result = CLD.detect_language(string)[:code] cld_result = CLD3::NNetLanguageIdentifier.new(0, 2048).find_language(string)
expect(cld_result).not_to eq :en expect(cld_result).not_to eq :en
result = described_class.new(string).to_iso_s result = described_class.new(string).to_iso_s