We already know about one regex limitation, which is that they cannot
segment words in e.g. Japanese, Chinese, or Thai. It may also end up
that regex matching is too slow compared to other methods.
However, the regex is an implementation detail. We still want the
ability to switch between "occurs anywhere" and "match whole word", and
caching the matcher result is likely to still be important (since the
matcher itself won't change nearly as often as status ingress rate).
Therefore, we ought to be able to change the cache keys to reflect a
change of data structure.
(Old cache keys expire within minutes, so they shouldn't be too big of
an issue. Old cache keys could also be explicitly removed by an
instance administrator.)
It is reasonable to expect someone to enter #foo to mute hashtag #foo.
However, tags are recorded on statuses without the preceding #.
To adjust for this, we build a separate tag matcher and use
Tag::HASHTAG_RE to extract a hashtag from the hashtag syntax.
When given two regexps, Regexp.union preserves the options set (or not
set) on each regex; this meant that none of the multiline (m),
case-insensitivity (i), or extended syntax (x) options were set. Our
regexps are written expecting the m, i, and x options were set on all of
them, so we need to make sure that we preserve that behavior.
Note that this will only hide/show *future* reblogs by a user, and does
nothing to remove/add reblogs that are already in the timeline. I don't
think that's a particularly confusing behavior, and it's a lot easier
to implement (similar to mutes, I believe).
We have changed how we store reblogs in the redis for bigint IDs. This process is done by 1) scan all entries in users feed, and 2) re-store reblogs by 3 write commands.
However, this operation is really slow for large instances. e.g. 1hrs on friends.nico (w/ 50k users). So I have tried below tweaks.
* It checked non-reblogs by `entry[0] == entry[1]`, but this condition won't work because `entry[0]` is String while `entry[1]` is Float. Changing `entry[0].to_i == entry[1]` seems work.
-> about 4-20x faster (feed with less reblogs will be faster)
* Write operations can be batched by pipeline
-> about 6x faster
* Wrap operation by Lua script and execute by EVALSHA command. This really reduces packets between Ruby and Redis.
-> about 3x faster
I've taken Lua script way, though doing other optimizations may be enough.
Glitch::KeywordMute's name is inferred as glitch_keyword_mutes, and in
templates this turns into e.g. settings/glitch/keyword_mutes. Going
along with this convention means a lot of file movement, though, and for
a UI that's as temporary and awkward as this one I think it's less
effort to slap a bunch of as: options everywhere.
We'll do the Right Thing when we build out the API and frontend UI.
Also make the keyword-building methods private: they always probably
should have been private, but now I have encoded enough fun and games
into them that it now seems wrong for them to *not* be private.
It is possible to cache a Regexp object, but I'm not sure what happens
if e.g. that object remains in cache across two different Ruby versions.
Caching a string seems to raise fewer questions.
David Yip