A new rake task emojis:generate downloads a full list of valid
unicode sequences from unicode.org and checks it against existing
Twemoji files, finally generating a map from each sequence to the
existing file (e.g. when there's multiple ways an emoji can be
expressed). The map is dumped into app/javascript/mastodon/emoji_map.json
That file is loaded by emojione_light.js (now a misnomer) which
decorates it further with shortcodes taken from emoji-mart's index.
* Add emoji autosuggest
Some credit goes to glitch-soc/mastodon#149
* Remove server-side shortcode->unicode conversion
* Insert shortcode when suggestion is custom emoji
* Remove remnant of server-side emojis
* Update style of autosuggestions
* Fix wrong emoji filenames generated in autosuggest item
* Do not lazy load emoji picker, as that no longer works
* Fix custom emoji autosuggest
* Fix multiple "Custom" categories getting added to emoji index, only add once
* Fix JavaScript interface with long IDs
Somewhat predictably, the JS interface handled IDs as numbers, which in
JS are IEEE double-precision floats. This loses some precision when
working with numbers as large as those generated by the new ID scheme,
so we instead handle them here as strings. This is relatively simple,
and doesn't appear to have caused any problems, but should definitely
be tested more thoroughly than the built-in tests. Several days of use
appear to support this working properly.
BREAKING CHANGE:
The major(!) change here is that IDs are now returned as strings by the
REST endpoints, rather than as integers. In practice, relatively few
changes were required to make the existing JS UI work with this change,
but it will likely hit API clients pretty hard: it's an entirely
different type to consume. (The one API client I tested, Tusky, handles
this with no problems, however.)
Twitter ran into this issue when introducing Snowflake IDs, and decided
to instead introduce an `id_str` field in JSON responses. I have opted
to *not* do that, and instead force all IDs to 64-bit integers
represented by strings in one go. (I believe Twitter exacerbated their
problem by rolling out the changes three times: once for statuses, once
for DMs, and once for user IDs, as well as by leaving an integer ID
value in JSON. As they said, "If you’re using the `id` field with JSON
in a Javascript-related language, there is a very high likelihood that
the integers will be silently munged by Javascript interpreters. In most
cases, this will result in behavior such as being unable to load or
delete a specific direct message, because the ID you're sending to the
API is different than the actual identifier associated with the
message." [1]) However, given that this is a significant change for API
users, alternatives or a transition time may be appropriate.
1: https://blog.twitter.com/developer/en_us/a/2011/direct-messages-going-snowflake-on-sep-30-2011.html
* Additional fixes for stringified IDs in JSON
These should be the last two. These were identified using eslint to try
to identify any plain casts to JavaScript numbers. (Some such casts are
legitimate, but these were not.)
Adding the following to .eslintrc.yml will identify casts to numbers:
~~~
no-restricted-syntax:
- warn
- selector: UnaryExpression[operator='+'] > :not(Literal)
message: Avoid the use of unary +
- selector: CallExpression[callee.name='Number']
message: Casting with Number() may coerce string IDs to numbers
~~~
The remaining three casts appear legitimate: two casts to array indices,
one in a server to turn an environment variable into a number.
* Back out RelationshipsController Change
This was made to make a test a bit less flakey, but has nothing to
do with this branch.
* Change internal streaming payloads to stringified IDs as well
Per
https://github.com/tootsuite/mastodon/pull/5019#issuecomment-330736452
we need these changes to send deleted status IDs as strings, not
integers.
* Custom emoji
- In OStatus: `<link rel="emoji" name="coolcat" href="http://..." />`
- In ActivityPub: `{ type: "Emoji", name: ":coolcat:", href: "http://..." }`
- In REST API: Status object includes `emojis` array (`shortcode`, `url`)
- Domain blocks with reject media stop emojis
- Emoji file up to 50KB
- Web UI handles custom emojis
- Static pages render custom emojis as `<img />` tags
Side effects:
- Undo #4500 optimization, as I needed to modify it to restore
shortcode handling in emojify()
- Formatter#plaintext should now make sure stripped out line-breaks
and paragraphs are replaced with newlines
* Fix emoji at the start not being converted
* Fix ActivityPub handling of replies when LOCAL_DOMAIN ≠ WEB_DOMAIN (#4895)
For all intents and purposes, `local_url?` is used to check if an URL refers
to the Web UI or the various API endpoints of the local instances. Those things
reside on `WEB_DOMAIN` and not `LOCAL_DOMAIN`.
* Change local_url? spec, as all URLs handled by Mastodon are based on WEB_DOMAIN
When a new user confirms their e-mail, bootstrap their home timeline
by automatically following a set of accounts. By default, all local
admin accounts (that are unlocked). Can be customized by new admin
setting (comma-separated usernames, local and unlocked only)
In before, the method uses stream_entry id as status id, so replied status was wrongly selected.
This PR uses StatusFinder which was introduced with `Api::Web::EmbedsController`.
* Fix language filter codes
CLD3 returns BCP-47 language identifier, filter settings expect
identifiers in the ISO 639-1 format. Convert between formats,
and exclude duplicate languages from filter choices (zh-CN->zh)
* Fix zh name
* Decouple Status#local? from uri being nil
* Replace on-the-fly URI generation with stored URIs
- Generate URI in after_save hook for local statuses
- Use static value in TagManager when available, fallback to tag format
- Make TagManager use ActivityPub::TagManager to understand new format
- Adjust tests
* Use other heuristic for locality of old statuses, do not perform long query
* Exclude tombstone stream entries from Atom feed
* Prevent nil statuses from landing in Pubsubhubbub::DistributionWorker
* Fix URI not being saved (#4818)
* Add more specs for Status
* Save generated uri immediately
and also fix method order to minimize diff.
* Fix alternate HTML URL in Atom
* Fix tests
* Remove not-null constraint from statuses migration to speed it up
* Raise an error for remote url in StatusFinder
Previous implementation had allowed remote url with status id which also exists on local.
Then that bug leads /api/web/embed to return wrong embed url.
* Fix oembed_controller_spec
* Make PreviewCard records reuseable between statuses
**Warning!** Migration truncates preview_cards tablec
* Allow a wider thumbnail for link preview, display it in horizontal layout (#4648)
* Delete preview cards files before truncating
* Rename old table instead of truncating it
* Add mastodon:maintenance:remove_deprecated_preview_cards
* Ignore deprecated_preview_cards in schema definition
* Fix null behaviour
- Use statuses controller for embeds instead of stream entries controller
- Prefer /@:username/:id/embed URL for embeds
- Use /@:username as author_url in OEmbed
- Add follow link to embeds which opens web intent in new window
- Use redis cache in development
- Cache entire embed
Requires moving Atom rendering from DistributionWorker (where
`stream_entry.status` is already nil) to inline (where
`stream_entry.status.destroyed?` is true) and distributing that.
Unfortunately, such XML renderings can no longer be easily chained
together into one payload of n items.
* Add handling of Linked Data Signatures in payloads
* Add a way to sign JSON, fix canonicalization of signature options
* Fix signatureValue encoding, send out signed JSON when distributing
* Add missing security context
* Allow multiple pinned statuses to be shown and make them be ordered by pinned date
* Set timestamps NOT NULL
* Make single-line pinned_statuses
* Spec for pinned_statuses
* Remove redundant empty line
* Process Create / Announce activity in FetchRemoteStatusService
* Use activity URL in ActivityPub for reblogs
* Redirect to the original status on StatusesController#show
* Add code for creating/managing apps to settings section
* Add specs for app changes
* Fix controller spec
* Fix view file I pasted over by mistake
* Add locale strings. Add 'my apps' to nav
* Add Client ID/Secret to App page. Add some visual separation
* Fix rubocop warnings
* Fix embarrassing typo
I lost an `end` statement while fixing a merge conflict.
* Add code for creating/managing apps to settings section
- Add specs for app changes
- Add locale strings. Add 'my apps' to nav
- Add Client ID/Secret to App page. Add some visual separation
- Fix some bugs/warnings
* Update to match code standards
* Trigger notification
* Add warning about not sharing API secrets
* Tweak spec a bit
* Cleanup fixture creation by using let!
* Remove unused key
* Add foreign key for application<->user
* Fallback to OStatus in FetchAtomService
* Skip activity+json link if that activity is Person without inbox
* If unsupported activity was detected and all other URLs failed, retry with ActivityPub-less Accept header
* Allow mention to OStatus account in ActivityPub
* Don't update profile with inbox-less Person object
*Note: OStatus URIs are invalid for ActivityPub. But we have them for
as long as we want to keep old OStatus-sourced content and as long as
we remain OStatus-compatible.*
- In Announce handling, if object URI is not a URL, fallback to object URL
- Do not use specialized ThreadResolveWorker, rely on generalized handling
- When serializing notes, if parent's URI is not a URL, use parent's URL
* Add ActivityPub inbox
* Handle ActivityPub deletes
* Handle ActivityPub creates
* Handle ActivityPub announces
* Stubs for handling all activities that need to be handled
* Add ActivityPub actor resolving
* Handle conversation URI passing in ActivityPub
* Handle content language in ActivityPub
* Send accept header when fetching actor, handle JSON parse errors
* Test for ActivityPub::FetchRemoteAccountService
* Handle public key and icon/image when embedded/as array/as resolvable URI
* Implement ActivityPub::FetchRemoteStatusService
* Add stubs for more interactions
* Undo activities implemented
* Handle out of order activities
* Hook up ActivityPub to ResolveRemoteAccountService, handle
Update Account activities
* Add fragment IDs to all transient activity serializers
* Add tests and fixes
* Add stubs for missing tests
* Add more tests
* Add more tests
* Refactored Avatar and AvatarOverlay (DRY) to have 'account' as prop.
Also removed animate attribute from compose navigation bar, which should
have never been there. Added test for avatar overlay.
* fix broken tests
* god dammit another bug in tests! travis please let this pass
* formatting in avatar overlay
* fix(dropdown_menu): Open as modal on mobile
* fix(dropdown_menu): Open modal on touch
* fix(dropdown_menu): Show status
* fix(dropdown_menu): Max dimensions and reduce padding
* chore(dropdown_menu): Test new functionality
* refactor: Use DropdownMenuContainer instead of DropdownMenu
* feat(privacy_dropdown): Open as modal on touch devices
* feat(modal_root): Do not load actions-modal async
* Use the same emoji data on the frontend and backend
* Move emoji.json to repository, add tests
This way you don't need to install node dependencies if you only
want to run Ruby code
* Do not raise unretryable exceptions in ResolveRemoteAccountService
* Removed fatal exceptions from ResolveRemoteAccountService
Exceptions that cannot be retried should not be raised. New exception
class for those that can be retried (Mastodon::UnexpectedResponseError)
* Refactor ResolveRemoteAccountService
* Remove trailing whitespace
* Use redis locks around critical ResolveRemoteAccountService code
* Add test for race condition of lock
* Wrap methods of ProcessFeedService::ProcessEntry in classes
This is a change same with 425acecfdb, except
that it has the following changes:
* Revert irrelevant change in find_or_create_conversation
* Fix error handling for RemoteActivity
* Introduce Ostatus name space
* Improve webfinger templates and make tests more flexible
* Clean up AS2 representation of actor
* Refactor outbox
* Create activities representation
* Add representations of followers/following collections, do not redirect /users/:username route if format is empty
* Remove unused translations
* ActivityPub endpoint for single statuses, add ActivityPub::TagManager for better
URL/URI generation
* Add ActivityPub::TagManager#to
* Represent all attachments as Document instead of Image/Video specifically
(Because for remote ones we may not know for sure)
Add mentions and hashtags representation to AP notes
* Add AP-resolvable hashtag URIs
* Use ActiveModelSerializers for ActivityPub
* Clean up unused translations
* Separate route for object and activity
* Adjust cc/to matrices
* Add to/cc to activities, ensure announce activity embeds target status and
not the wrapper status, add "id" to all collections
* Add Request class with HTTP signature generator
Spec: https://tools.ietf.org/html/draft-cavage-http-signatures-06
* Add HTTP signature verification concern
* Add test for SignatureVerification concern
* Add basic test for Request class
* Make PuSH subscribe/unsubscribe requests use new Request class
Accidentally fix lease_seconds not being set and sent properly, and
change the new minimum subscription duration to 1 day
* Make all PuSH workers use new Request class
* Make Salmon sender use new Request class
* Make FetchLinkService use new Request class
* Make FetchAtomService use the new Request class
* Make Remotable use the new Request class
* Make ResolveRemoteAccountService use the new Request class
* Add more tests
* Allow +-30 seconds window for signed request to remain valid
* Disable time window validation for signed requests, restore 7 days
as PuSH subscription duration (which was previous default due to a bug)
* chore(yarn): Install babel-plugin-preval as development dependency
* feat(babel): Add preval as a plugin
* feat(emojione_light): Prevaled module what tree-shaked emojione
* refactor(emoji): Use emojione_light
* feat: Preload emojione_picker bundle
* fix(emojione_light): Do not use Object.entries
* fix(emojify): Update tests
* chore(emojione_light): Remove silly ascii art
- Use unicode when selecting emoji through picker
- Convert shortcodes to unicode when storing text input server-side
- Do not convert shortcodes in JS anymore
* Redesign the landing page, mount public timeline on it
* Adjust the standalone mounted component to the lacking of router
* Adjust auth layout pages to new design
* Fix tests
* Standalone public timeline polling every 5 seconds
* Remove now obsolete translations
* Add responsive design for new landing page
* Address reviews
* Add floating clouds behind frontpage form
* Use access token from public page when available
* Fix mentions and hashtags links, cursor on status content in standalone mode
* Add footer link to source code
* Fix errors on pages that don't embed the component, use classnames
* Fix tests
* Change anonymous autoPlayGif default to false
* When gif autoplay is disabled, hover to play
* Add option to hide the timeline preview
* Slightly improve alt layout
* Add elephant friend to new frontpage
* Display "back to mastodon" in place of "login" when logged in on frontpage
* Change polling time to 3s
* Added a success page to remote following
Includes follow-through links to web (the old redirect target) and back to the remote user's profile
* Use Account.new in spec instead of a fake with only id
(fixes spec)
* Fabricate(:account) over Account.new
* Remove self from the success text
(and all HTML with it)
* Specs for language detection
* Use CharlockHolmes instead of NKF
* Correct mistakes
* Correct style
* Set hint_enc instead of falling back and strip_tags
* Improve specs
* Add dependencies
* add a system_font_ui setting on the server
* Plug the system_font_ui on the front-end
* add EN/FR locales for the new setting
* put Roboto after all other fonts
* remove trailing whitespace so CodeClimate is happy
* fix user_spec.rb
* correctly write user_spect this time
* slightly better way of adding the classes
* add comments to the system-font stack for clarification
* use .system-font for the class instead
* don't use multiple lines for comments
* remove trailing whitespace
* use the classnames module for consistency
* use `mastodon-font-sans-serif` instead of Roboto directly
In from_redis method, statuses retrieved from the database was mapped to
the IDs retrieved from Redis. It was equivalent to order from high to low
because those IDs are sorted in the same order.
Statuses are ordered with the ID by default, so we do not have to reorder.
Sorting statuses in the database is even faster since the IDs are indexed
with B-tree.
* Fix#3910 - Require OTP authentication to disable 2FA. Also, remove ability
to generate new OTP backup codes *after* initial backup codes were handed
out during activation
* Restore recovery code re-generation
* Improve display of some 2FA elements
* Add overview of active sessions
* Better display of browser/platform name
* Improve how browser information is stored and displayed for sessions overview
* Fix test
* Introduce domains method to Account relation
Account had followers_domains method, which was excessively specific.
Let relation of Account have domains method instead.
* Move follow_mapping in Account to AccountInteractions
* Introduce shared examples for AccountAvatar inclusion
* Cover Account more
* Fix regression from #3842
Simplify the query by omitting all direct statuses. Private statuses
are allowed because they are from accounts we are following (so
by definition)
Resolves#3887 (alternative)
* Adjust test
The classes using Status.as_home_timeline, namely Feed and
PrecomputeFeedService are expected to filter direct statuses as
FanOutWriteService does, but their filtering were incomplete or missing.
This commit solves the problem by filtering direct statuses in
as_home_timeline as the other similar methods such as as_public_timeline
does.
* Whitelist allowed classes for federated statuses
Allowed classes are currently:
- Any microformats class (h/p/u/dt/e-*)
- the classes mention, hashtag, ellipses and invisible.
this last one is somewhat suspect, but Mastodon currently uses it to render hidden link text.
resolved#3790
* Fix code style
* Make Pubsubhubbub::DistributionWorker handle both single stream entry
arguments, as well as arrays of stream entries
* Add BatchedRemoveStatusService, make SuspendAccountService use it
* Improve method names
* Add test
* Add more tests
* Use PuSH payloads of 100 to have a clear mapping of
1000 input statuses -> 10 PuSH payloads
It was nice while it lasted
* Add form for account deletion
* If avatar or header are gone from source, remove them
* Add option to have SuspendAccountService remove user record, add tests
* Exclude suspended accounts from search
* Fix#2619 - When redis feed is empty, fall back to database
* Use redis value to return feed from database only while RegenerationWorker
hasn't finished running
* Fix specs
* Replace usage of reject!
When case insensitivity is enabled via devise's `config.case_insensitive_keys` then `.find_for_authentication` method needs to be used instead of `.find_by` because second mentioned returns `nil` when valid email with different cases is passed.
More info https://github.com/plataformatec/devise/wiki/How-To:-Use-case-insensitive-emails
* Improve default language decision
This change allows to takes account of accepted language determined by
the user agent even if the custom default locale of the instance is
configured.
* Cover Localized more
* Fix code style
Each of mute, favourite, reblog has been updated to:
- Have a separate controller with just a create and destroy action
- Preserve historical route names to not break the API
- Mild refactoring to break up long methods
* Add specs for api statuses routes
* Update favourited_by and reblogged_by api routes
* Move methods into new controllers
* Use load_accounts methods to simplify index actions
* Clean up load_accounts methods
* Clean up link header generation
* Check for link headers in specs
* Remove unused actions from api/v1/statuses controller
* Remove specs for moved actions
* Move ApiController to Api/BaseController
* API controllers inherit from Api::BaseController
* Add coverage for various error cases in api/base controller
* Coverage for rate limit headers
* Move rate limit headers methods to concern
* Move throttle check to condition on before_action
* Move match_data variable into method
* Move utc timestamp to separate method
* Move header setting into smaller methods
* specs cleanup
* Sort results by the name
* Switch search method to simple `LIKE` matching instead of tsvector/tsquery
Previously we used scores from ts_rank_cd() to sort results, but it didn't work
because the function returns same score for all results. It's not for calculate
similarity of single words. Sometimes this bug even push out exact matching tag
from results.
Additionally, PostgreSQL supports prefix searching with standard btree index.
Using it offers simpler code, but also less index size and some speed.
* Add a StatusFilter class to identify visibility of statuses by accounts
* Extract StatusThreadingConcern from Status
* Clarify purpose of checking for nil account
* Redirect to streaming_api_base_url
When Rails receives a request to streaming API, it most likely
means that there is another host which is configured to respond
to it. This is to redirect clients to that host if
`STREAMING_API_BASE_URL` is set as another host.
* Use the new Ruby 1.9 hash syntax
* Do not fall back to StreamEntry if object_type is unavailable in TagManager
Since 6d6a429af8, when Status, the only model
with stream_entry, and StreamEntry got its own logic in uri_for and
url_for, the purpose of the fallbacks to activity_type of StreamEntry
became unclear.
This commit removes the fallbacks. When adding another model with
stream_entry in future, consider to update uri_for and url_for.
* Cover TagManager more
* Do not default the format in ProviderDiscovery
The format should be determined when discovering, as it is in the current
implementation, and it is a flaw if it is not determined.
* Spec ProviderDiscovery
* Add failing specs for hashtag and username extraction in language detector
* Remove usernames and hashtags from text before language detection
* Handle multiple instances of special case, and reduce whitespace
* Move specs for account finder methods to concern spec
* Move account finder methods to concern
* Improve spec wording
* Use more explicit comparison to ensure correct return value
* Add coverage for .find_local! and .find_remote!
* Add some methods to the finder
* Use arel on matching_username method
* Avoid ternary in matching domain method
* Simplify finder methods
* Use an AccountFinder class to simplify lookup
Eugen Rochko