Commit Graph

17 Commits (b5d60a90a32ee9698f271c5a64d3fbf0405ec4ca)

Author SHA1 Message Date
Eugen Rochko 4dc87ffc06 Add support for structured data and more OpenGraph tags to link cards ()
Save preview cards under their canonical URL

Increase max redirects to follow from 2 to 3
2021-11-05 23:23:05 +01:00
Claire 53d99e7426 Fix URL scanning in note length validator and preview card fetching ()
* Add tests

* Fix URL scanning in note length validator and preview card fetching
2021-03-04 00:12:26 +01:00
ThibG 32ef133fa6 Fix FetchLinkCardServices crashing on a tags without a target ()
* Add test for links without targets

* Fix FetchLinkCardServices crashing on a tags without a target
2019-11-21 16:04:52 +01:00
BSKY 483cf4d52a Add noopener and/or noreferrer () 2019-10-24 22:44:42 +02:00
Eugen Rochko 67713f8b08 Remove HEAD request from fetching link previews ()
It is not really necessary and we need to reduce requests
2019-10-01 04:54:10 +02:00
ThibG f76665a276 Ignore low-confidence CharlockHolmes guesses when parsing link cards ()
* Add failing test for windows-1251 link cards

* Ignore low-confidence CharlockHolmes guesses

Fixes 

* Fix no method error when charlock holmes cannot detect charset
2018-12-17 19:19:45 +01:00
Eugen Rochko ca1c696dbd Slightly reduce RAM usage ()
* No need to re-require sidekiq plugins, they are required via Gemfile

* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile

* Replace ruby-oembed with FetchOEmbedService

Reduce startup by 45382 allocated objects

* Remove preloaded JSON-LD in favour of caching HTTP responses

Reduce boot RAM by about 6 MiB

* Fix tests

* Fix test suite by stubbing out JSON-LD contexts
2018-05-02 18:58:48 +02:00
Daniel King 845ea13622 Fix URLs incorrectly having trailing hyphen removed ()
In cases where a URL has a trailing hyphen the FetchLinkCardService incorrectly removes the hyphen when it is parsed

The hyphen is not a reserved character in the URI spec https://tools.ietf.org/html/rfc3986#section-2.2
2018-02-11 23:49:18 +01:00
ふぁぼ原 c71727ca55 Enable to recognize most kinds of characters as URL paths () 2017-09-14 18:03:20 +02:00
Eugen Rochko 73c613dcfc Switch to static URIs, new URI format in both protocols for new statuses ()
* Decouple Status#local? from uri being nil

* Replace on-the-fly URI generation with stored URIs

- Generate URI in after_save hook for local statuses
- Use static value in TagManager when available, fallback to tag format
- Make TagManager use ActivityPub::TagManager to understand new format
- Adjust tests

* Use other heuristic for locality of old statuses, do not perform long query

* Exclude tombstone stream entries from Atom feed

* Prevent nil statuses from landing in Pubsubhubbub::DistributionWorker

* Fix URI not being saved ()

* Add more specs for Status

* Save generated uri immediately

and also fix method order to minimize diff.

* Fix alternate HTML URL in Atom

* Fix tests

* Remove not-null constraint from statuses migration to speed it up
2017-09-06 19:01:28 +02:00
Eugen Rochko e9e271878e Make PreviewCard records reuseable between statuses ()
* Make PreviewCard records reuseable between statuses

**Warning!** Migration truncates preview_cards tablec

* Allow a wider thumbnail for link preview, display it in horizontal layout ()

* Delete preview cards files before truncating

* Rename old table instead of truncating it

* Add mastodon:maintenance:remove_deprecated_preview_cards

* Ignore deprecated_preview_cards in schema definition

* Fix null behaviour
2017-09-01 16:20:16 +02:00
nullkal 07024f56df Use charlock_holmes instead of nkf at FetchLinkCardService ()
* Specs for language detection

* Use CharlockHolmes instead of NKF

* Correct mistakes

* Correct style

* Set hint_enc instead of falling back and strip_tags

* Improve specs

* Add dependencies
2017-07-08 22:44:31 +02:00
abcang 8041c97d52 Fix Nokogiri::HTML at FetchLinkCardService () 2017-07-05 14:54:21 +02:00
Eugen Rochko 93e9f8b3ed Fix - Resolve preview cards for remote statuses as well as local ones () 2017-05-17 00:41:15 +02:00
Eugen Rochko ee7719d540 Fix - Send HEAD request ahead of GET when fetching URL previews () 2017-05-10 23:30:07 +02:00
Eugen Rochko 4a7dc4fadc OEmbed support for PreviewCard ()
* OEmbed support for PreviewCard

* Improve ProviderDiscovery code failure treatment

* Do not crawl links if there is a content warning, since those
don't display a link card anyway

* Reset db schema

* Fresh migrate

* Fix rubocop style issues
Fix  - return existing access token when applicable instead of creating new

* Fix test

* Extract http client to helper

* Improve oembed controller
2017-04-27 14:42:22 +02:00
Eugen 87f7a3922c Punycode URI normalization ()
* Fix  - Whenever about to send a HTTP request, normalize the URI

* Add test for IDN request in FetchLinkCardService

* Perform IDN normalization on domains before they are stored in the DB
2017-04-25 02:47:31 +02:00