th: Merge remote-tracking branch 'glitch/main'

fixes: CVE-2023-36459
fixes: CVE-2023-36460
fixes: CVE-2023-36461
fixes: CVE-2023-36462
fixes: GHSA-55j9-c3mp-6fcq
fixes: GHSA-9928-3cp5-93fm
fixes: GHSA-9pxv-6qvf-pjwc
fixes: GHSA-ccm4-vgcc-73hp

CHANGELOG.md

@@ -2,6 +2,54 @@
 
 All notable changes to this project will be documented in this file.
 
+## [4.1.3] - 2023-07-06
+
+### Added
+
+- Add fallback redirection when getting a webfinger query `LOCAL_DOMAIN@LOCAL_DOMAIN` ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/23600))
+
+### Changed
+
+- Change OpenGraph-based embeds to allow fullscreen ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25058))
+- Change AccessTokensVacuum to also delete expired tokens ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/24868))
+- Change profile updates to be sent to recently-mentioned servers ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/24852))
+- Change automatic post deletion thresholds and load detection ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/24614))
+- Change `/api/v1/statuses/:id/history` to always return at least one item ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25510))
+- Change auto-linking to allow carets in URL query params ([renchap](https://github.com/mastodon/mastodon/pull/25216))
+
+### Removed
+
+- Remove invalid `X-Frame-Options: ALLOWALL` ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25070))
+
+### Fixed
+
+- Fix wrong view being displayed when a webhook fails validation ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25464))
+- Fix soft-deleted post cleanup scheduler overwhelming the streaming server ([ThisIsMissEm](https://github.com/mastodon/mastodon/pull/25519))
+- Fix incorrect pagination headers in `/api/v2/admin/accounts` ([danielmbrasil](https://github.com/mastodon/mastodon/pull/25477))
+- Fix multiple inefficiencies in automatic post cleanup worker ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/24607), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/24785), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/24840))
+- Fix performance of streaming by parsing message JSON once ([ThisIsMissEm](https://github.com/mastodon/mastodon/pull/25278), [ThisIsMissEm](https://github.com/mastodon/mastodon/pull/25361))
+- Fix CSP headers when `S3_ALIAS_HOST` includes a path component ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25273))
+- Fix `tootctl accounts approve --number N` not aproving N earliest registrations ([danielmbrasil](https://github.com/mastodon/mastodon/pull/24605))
+- Fix reports not being closed when performing batch suspensions ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/24988))
+- Fix being able to vote on your own polls ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25015))
+- Fix race condition when reblogging a status ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25016))
+- Fix “Authorized applications” inefficiently and incorrectly getting last use date ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25060))
+- Fix “Authorized applications” crashing when listing apps with certain admin API scopes ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25713))
+- Fix multiple N+1s in ConversationsController ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25134), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/25399), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/25499))
+- Fix user archive takeouts when using OpenStack Swift ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/24431))
+- Fix searching for remote content by URL not working under certain conditions ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25637))
+- Fix inefficiencies in indexing content for search ([VyrCossont](https://github.com/mastodon/mastodon/pull/24285), [VyrCossont](https://github.com/mastodon/mastodon/pull/24342))
+
+### Security
+
+- Add finer permission requirements for managing webhooks ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25463))
+- Update dependencies
+- Add hardening headers for user-uploaded files ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/25756))
+- Fix verified links possibly hiding important parts of the URL (CVE-2023-36462)
+- Fix timeout handling of outbound HTTP requests (CVE-2023-36461)
+- Fix arbitrary file creation through media processing (CVE-2023-36460)
+- Fix possible XSS in preview cards (CVE-2023-36459)
+
 ## [4.1.2] - 2023-04-04
 
 ### Fixed

app/controllers/api/v2/search_controller.rb

@@ -34,11 +34,11 @@ class Api::V2::SearchController < Api::BaseController
       params[:q],
       current_account,
       limit_param(RESULTS_LIMIT),
-      search_params.merge(resolve: truthy_param?(:resolve), exclude_unreviewed: truthy_param?(:exclude_unreviewed))
+      search_params.merge(resolve: truthy_param?(:resolve), exclude_unreviewed: truthy_param?(:exclude_unreviewed), following: truthy_param?(:following))
     )
   end
 
   def search_params
-    params.permit(:type, :offset, :min_id, :max_id, :account_id)
+    params.permit(:type, :offset, :min_id, :max_id, :account_id, :following)
   end
 end

app/helpers/formatting_helper.rb

@@ -64,6 +64,10 @@ module FormattingHelper
   end
 
   def account_field_value_format(field, with_rel_me: true)
+    if field.verified? && !field.account.local?
+      TextFormatter.shortened_link(field.value_for_verification)
+    else
       html_aware_format(field.value, field.account.local?, with_rel_me: with_rel_me, with_domains: true, multiline: false)
     end
   end
+end

app/javascript/flavours/glitch/components/poll.jsx

@@ -131,6 +131,10 @@ class Poll extends ImmutablePureComponent {
     this.props.refresh();
   };
 
+  handleReveal = () => {
+    this.setState({ revealed: true });
+  }
+
   renderOption (option, optionIndex, showResults) {
     const { poll, lang, disabled, intl } = this.props;
     const pollVotesCount  = poll.get('voters_count') || poll.get('votes_count');
@@ -206,14 +210,14 @@ class Poll extends ImmutablePureComponent {
 
   render () {
     const { poll, intl } = this.props;
-    const { expired } = this.state;
+    const { revealed, expired } = this.state;
 
     if (!poll) {
       return null;
     }
 
     const timeRemaining = expired ? intl.formatMessage(messages.closed) : <RelativeTimestamp timestamp={poll.get('expires_at')} futureDate />;
-    const showResults   = poll.get('voted') || expired;
+    const showResults   = poll.get('voted') || revealed || expired;
     const disabled      = this.props.disabled || Object.entries(this.state.selected).every(item => !item);
 
     let votesCount = null;
@@ -232,9 +236,10 @@ class Poll extends ImmutablePureComponent {
 
         <div className='poll__footer'>
           {!showResults && <button className='button button-secondary' disabled={disabled || !this.context.identity.signedIn} onClick={this.handleVote}><FormattedMessage id='poll.vote' defaultMessage='Vote' /></button>}
-          {showResults && !this.props.disabled && <span><button className='poll__link' onClick={this.handleRefresh}><FormattedMessage id='poll.refresh' defaultMessage='Refresh' /></button> · </span>}
+          {!showResults && <><button className='poll__link' onClick={this.handleReveal}><FormattedMessage id='poll.reveal' defaultMessage='See results' /></button> · </>}
+          {showResults && !this.props.disabled && <><button className='poll__link' onClick={this.handleRefresh}><FormattedMessage id='poll.refresh' defaultMessage='Refresh' /></button> · </>}
           {votesCount}
-          {poll.get('expires_at') && <span> · {timeRemaining}</span>}
+          {poll.get('expires_at') && <> · {timeRemaining}</>}
         </div>
       </div>
     );

app/javascript/flavours/glitch/components/status_content.jsx

@@ -163,7 +163,7 @@ class StatusContent extends PureComponent {
 
       if (mention) {
         link.addEventListener('click', this.onMentionClick.bind(this, mention), false);
-        link.setAttribute('title', mention.get('acct'));
+        link.setAttribute('title', `@${mention.get('acct')}`);
         if (rewriteMentions !== 'no') {
           while (link.firstChild) link.removeChild(link.firstChild);
           link.appendChild(document.createTextNode('@'));

app/javascript/flavours/glitch/features/account/components/header.jsx

@@ -398,6 +398,7 @@ class Header extends ImmutablePureComponent {
         <Helmet>
           <title>{titleFromAccount(account)}</title>
           <meta name='robots' content={(isLocal && isIndexable) ? 'all' : 'noindex'} />
+          <link rel='canonical' href={account.get('url')} />
         </Helmet>
       </div>
     );

app/javascript/flavours/glitch/features/firehose/index.jsx

@@ -103,7 +103,7 @@ const Firehose = ({ feedType, multiColumn }) => {
     (maxId) => {
       switch(feedType) {
       case 'community':
-        dispatch(expandCommunityTimeline({ onlyMedia }));
+        dispatch(expandCommunityTimeline({ maxId, onlyMedia }));
         break;
       case 'public':
         dispatch(expandPublicTimeline({ maxId, onlyMedia, allowLocalOnly }));
@@ -157,7 +157,8 @@ const Firehose = ({ feedType, multiColumn }) => {
     <DismissableBanner id='public_timeline'>
       <FormattedMessage
         id='dismissable_banner.public_timeline'
-       defaultMessage='These are the most recent public posts from people on this and other servers of the decentralized network that this server knows about.'
+        defaultMessage='These are the most recent public posts from people on the social web that people on {domain} follow.'
+        values={{ domain }}
       />
     </DismissableBanner>
   );
@@ -190,11 +191,11 @@ const Firehose = ({ feedType, multiColumn }) => {
       <div className='scrollable scrollable--flex'>
         <div className='account__section-headline'>
           <NavLink exact to='/public/local'>
-            <FormattedMessage tagName='div' id='firehose.local' defaultMessage='Local' />
+            <FormattedMessage tagName='div' id='firehose.local' defaultMessage='This server' />
           </NavLink>
 
           <NavLink exact to='/public/remote'>
-            <FormattedMessage tagName='div' id='firehose.remote' defaultMessage='Remote' />
+            <FormattedMessage tagName='div' id='firehose.remote' defaultMessage='Other servers' />
           </NavLink>
 
           <NavLink exact to='/public'>

app/javascript/flavours/glitch/features/public_timeline/index.jsx

@@ -13,6 +13,7 @@ import { expandPublicTimeline } from 'flavours/glitch/actions/timelines';
 import Column from 'flavours/glitch/components/column';
 import ColumnHeader from 'flavours/glitch/components/column_header';
 import DismissableBanner from 'flavours/glitch/components/dismissable_banner';
+import { domain } from 'flavours/glitch/initial_state';
 import StatusListContainer from 'flavours/glitch/features/ui/containers/status_list_container';
 
 import ColumnSettingsContainer from './containers/column_settings_container';
@@ -147,7 +148,7 @@ class PublicTimeline extends PureComponent {
         </ColumnHeader>
 
         <StatusListContainer
-          prepend={<DismissableBanner id='public_timeline'><FormattedMessage id='dismissable_banner.public_timeline' defaultMessage='These are the most recent public posts from people on this and other servers of the decentralized network that this server knows about.' /></DismissableBanner>}
+          prepend={<DismissableBanner id='public_timeline'><FormattedMessage id='dismissable_banner.public_timeline' defaultMessage='These are the most recent public posts from people on the social web that people on {domain} follow.' values={{ domain }} /></DismissableBanner>}
           timelineId={`public${onlyRemote ? ':remote' : (allowLocalOnly ? ':allow_local_only' : '')}${onlyMedia ? ':media' : ''}`}
           onLoadMore={this.handleLoadMore}
           trackScroll={!pinned}

app/javascript/flavours/glitch/features/status/index.jsx

@@ -771,6 +771,7 @@ class Status extends ImmutablePureComponent {
         <Helmet>
           <title>{titleFromStatus(intl, status)}</title>
           <meta name='robots' content={(isLocal && isIndexable) ? 'all' : 'noindex'} />
+          <link rel='canonical' href={status.get('url')} />
         </Helmet>
       </Column>
     );

app/javascript/flavours/glitch/styles/admin.scss

@@ -128,7 +128,6 @@ $content-width: 840px;
         }
 
         &.selected {
-          background: darken($ui-base-color, 2%);
           border-radius: 4px 0 0;
         }
       }
@@ -146,13 +145,9 @@ $content-width: 840px;
 
       .simple-navigation-active-leaf a {
         color: $primary-text-color;
-        background-color: darken($ui-highlight-color, 2%);
+        background-color: $ui-highlight-color;
         border-bottom: 0;
         border-radius: 0;
-
-        &:hover {
-          background-color: $ui-highlight-color;
-        }
       }
     }
 
@@ -246,12 +241,6 @@ $content-width: 840px;
             font-weight: 700;
             color: $primary-text-color;
             background: $ui-highlight-color;
-
-            &:hover,
-            &:focus,
-            &:active {
-              background: lighten($ui-highlight-color, 4%);
-            }
           }
         }
       }

app/javascript/flavours/glitch/styles/components/misc.scss

@@ -38,11 +38,11 @@
 }
 
 .button {
-  background-color: darken($ui-highlight-color, 3%);
+  background-color: $ui-button-background-color;
   border: 10px none;
   border-radius: 4px;
   box-sizing: border-box;
-  color: $primary-text-color;
+  color: $ui-button-color;
   cursor: pointer;
   display: inline-block;
   font-family: inherit;
@@ -62,14 +62,14 @@
   &:active,
   &:focus,
   &:hover {
-    background-color: $ui-highlight-color;
+    background-color: $ui-button-focus-background-color;
   }
 
   &--destructive {
     &:active,
     &:focus,
     &:hover {
-      background-color: $error-red;
+      background-color: $ui-button-destructive-focus-background-color;
       transition: none;
     }
   }
@@ -79,43 +79,22 @@
     cursor: default;
   }
 
-  &.button-alternative {
-    color: $inverted-text-color;
-    background: $ui-primary-color;
-
-    &:active,
-    &:focus,
-    &:hover {
-      background-color: lighten($ui-primary-color, 4%);
-    }
-  }
-
-  &.button-alternative-2 {
-    background: $ui-base-lighter-color;
-
-    &:active,
-    &:focus,
-    &:hover {
-      background-color: lighten($ui-base-lighter-color, 4%);
-    }
-  }
-
   &.button-secondary {
     font-size: 16px;
     line-height: 36px;
     height: auto;
-    color: $darker-text-color;
+    color: $ui-button-secondary-color;
     text-transform: none;
     background: transparent;
     padding: 6px 17px;
-    border: 1px solid lighten($ui-base-color, 12%);
+    border: 1px solid $ui-button-secondary-border-color;
 
     &:active,
     &:focus,
     &:hover {
-      background: lighten($ui-base-color, 4%);
+      border-color: $ui-button-secondary-focus-background-color;
-      border-color: lighten($ui-base-color, 16%);
+      color: $ui-button-secondary-focus-color;
-      color: lighten($darker-text-color, 4%);
+      background-color: $ui-button-secondary-focus-background-color;
       text-decoration: none;
     }
 
@@ -127,14 +106,14 @@
   &.button-tertiary {
     background: transparent;
     padding: 6px 17px;
-    color: $highlight-text-color;
+    color: $ui-button-tertiary-color;
-    border: 1px solid $highlight-text-color;
+    border: 1px solid $ui-button-tertiary-border-color;
 
     &:active,
     &:focus,
     &:hover {
-      background: $ui-highlight-color;
+      background-color: $ui-button-tertiary-focus-background-color;
-      color: $primary-text-color;
+      color: $ui-button-tertiary-focus-color;
       border: 0;
       padding: 7px 18px;
     }

app/javascript/flavours/glitch/styles/components/modal.scss

@@ -718,15 +718,15 @@
   }
 
   .button.button-secondary {
-    border-color: $inverted-text-color;
+    border-color: $ui-button-secondary-border-color;
-    color: $inverted-text-color;
+    color: $ui-button-secondary-color;
     flex: 0 0 auto;
 
     &:hover,
     &:focus,
     &:active {
-      border-color: lighten($inverted-text-color, 15%);
+      border-color: $ui-button-secondary-focus-background-color;
-      color: lighten($inverted-text-color, 15%);
+      color: $ui-button-secondary-focus-color;
     }
   }
 

app/javascript/flavours/glitch/styles/dashboard.scss

@@ -81,7 +81,7 @@
     display: flex;
     align-items: baseline;
     border-radius: 4px;
-    background: darken($ui-highlight-color, 2%);
+    background: $ui-button-background-color;
     color: $primary-text-color;
     transition: all 100ms ease-in;
     font-size: 14px;
@@ -94,7 +94,7 @@
     &:active,
     &:focus,
     &:hover {
-      background-color: $ui-highlight-color;
+      background-color: $ui-button-focus-background-color;
       transition: all 200ms ease-out;
     }
 

app/javascript/flavours/glitch/styles/forms.scss

@@ -512,8 +512,8 @@ code {
     width: 100%;
     border: 0;
     border-radius: 4px;
-    background: darken($ui-highlight-color, 2%);
+    background: $ui-button-background-color;
-    color: $primary-text-color;
+    color: $ui-button-color;
     font-size: 18px;
     line-height: inherit;
     height: auto;
@@ -535,7 +535,7 @@ code {
     &:active,
     &:focus,
     &:hover {
-      background-color: $ui-highlight-color;
+      background-color: $ui-button-focus-background-color;
     }
 
     &:disabled:hover {
@@ -543,15 +543,12 @@ code {
     }
 
     &.negative {
-      background: $error-value-color;
+      background: $ui-button-destructive-background-color;
-
-      &:hover {
-        background-color: lighten($error-value-color, 5%);
-      }
 
+      &:hover,
       &:active,
       &:focus {
-        background-color: darken($error-value-color, 5%);
+        background-color: $ui-button-destructive-focus-background-color;
       }
     }
   }

app/javascript/flavours/glitch/styles/mastodon-light/diff.scss

@@ -5,19 +5,6 @@ html {
   scrollbar-color: $ui-base-color rgba($ui-base-color, 0.25);
 }
 
-// Change the colors of button texts
-.button {
-  color: $white;
-
-  &.button-alternative-2 {
-    color: $white;
-  }
-
-  &.button-tertiary {
-    color: $highlight-text-color;
-  }
-}
-
 .simple_form .button.button-tertiary {
   color: $highlight-text-color;
 }
@@ -437,26 +424,6 @@ html {
   color: $white;
 }
 
-.button.button-tertiary {
-  &:hover,
-  &:focus,
-  &:active {
-    color: $white;
-  }
-}
-
-.button.button-secondary {
-  border-color: $darker-text-color;
-  color: $darker-text-color;
-
-  &:hover,
-  &:focus,
-  &:active {
-    border-color: darken($darker-text-color, 8%);
-    color: darken($darker-text-color, 8%);
-  }
-}
-
 .flash-message.warning {
   color: lighten($gold-star, 16%);
 }

app/javascript/flavours/glitch/styles/mastodon-light/variables.scss

@@ -7,6 +7,12 @@ $classic-primary-color: #9baec8;
 $classic-secondary-color: #d9e1e8;
 $classic-highlight-color: #6364ff;
 
+$blurple-600: #563acc; // Iris
+$blurple-500: #6364ff; // Brand purple
+$blurple-300: #858afa; // Faded Blue
+$grey-600: #4e4c5a; // Trout
+$grey-100: #dadaf3; // Topaz
+
 // Differences
 $success-green: lighten(#3c754d, 8%);
 
@@ -19,6 +25,13 @@ $ui-primary-color: #9bcbed;
 $ui-secondary-color: $classic-base-color !default;
 $ui-highlight-color: $classic-highlight-color !default;
 
+$ui-button-secondary-color: $grey-600 !default;
+$ui-button-secondary-border-color: $grey-600 !default;
+$ui-button-secondary-focus-color: $white !default;
+
+$ui-button-tertiary-color: $blurple-500 !default;
+$ui-button-tertiary-border-color: $blurple-500 !default;
+
 $primary-text-color: $black !default;
 $darker-text-color: $classic-base-color !default;
 $highlight-text-color: darken($ui-highlight-color, 8%) !default;

app/javascript/flavours/glitch/styles/variables.scss

@@ -1,10 +1,18 @@
 // Commonly used web colors
 $black: #000000; // Black
 $white: #ffffff; // White
-$success-green: #79bd9a; // Padua
+$red-600: #b7253d !default; // Deep Carmine
-$error-red: #df405a; // Cerise
+$red-500: #df405a !default; // Cerise
-$warning-red: #ff5050; // Sunset Orange
+$blurple-600: #563acc; // Iris
-$gold-star: #ca8f04; // Dark Goldenrod
+$blurple-500: #6364ff; // Brand purple
+$blurple-300: #858afa; // Faded Blue
+$grey-600: #4e4c5a; // Trout
+$grey-100: #dadaf3; // Topaz
+
+$success-green: #79bd9a !default; // Padua
+$error-red: $red-500 !default; // Cerise
+$warning-red: #ff5050 !default; // Sunset Orange
+$gold-star: #ca8f04 !default; // Dark Goldenrod
 
 $red-bookmark: $warning-red;
 
@@ -31,6 +39,22 @@ $ui-base-lighter-color: lighten(
 $ui-primary-color: $classic-primary-color !default; // Lighter
 $ui-secondary-color: $classic-secondary-color !default; // Lightest
 $ui-highlight-color: $classic-highlight-color !default;
+$ui-button-color: $white !default;
+$ui-button-background-color: $blurple-500 !default;
+$ui-button-focus-background-color: $blurple-600 !default;
+
+$ui-button-secondary-color: $grey-100 !default;
+$ui-button-secondary-border-color: $grey-100 !default;
+$ui-button-secondary-focus-background-color: $grey-600 !default;
+$ui-button-secondary-focus-color: $white !default;
+
+$ui-button-tertiary-color: $blurple-300 !default;
+$ui-button-tertiary-border-color: $blurple-300 !default;
+$ui-button-tertiary-focus-background-color: $blurple-600 !default;
+$ui-button-tertiary-focus-color: $white !default;
+
+$ui-button-destructive-background-color: $red-500 !default;
+$ui-button-destructive-focus-background-color: $red-600 !default;
 
 // Variables for texts
 $primary-text-color: $white !default;
@@ -39,6 +63,7 @@ $dark-text-color: $ui-base-lighter-color !default;
 $secondary-text-color: $ui-secondary-color !default;
 $highlight-text-color: lighten($ui-highlight-color, 8%) !default;
 $action-button-color: $ui-base-lighter-color !default;
+$action-button-focus-color: lighten($ui-base-lighter-color, 4%) !default;
 $passive-text-color: $gold-star !default;
 $active-passive-text-color: $success-green !default;
 

app/javascript/mastodon/components/poll.jsx

@@ -130,6 +130,10 @@ class Poll extends ImmutablePureComponent {
     this.props.refresh();
   };
 
+  handleReveal = () => {
+    this.setState({ revealed: true });
+  }
+
   renderOption (option, optionIndex, showResults) {
     const { poll, lang, disabled, intl } = this.props;
     const pollVotesCount  = poll.get('voters_count') || poll.get('votes_count');
@@ -205,14 +209,14 @@ class Poll extends ImmutablePureComponent {
 
   render () {
     const { poll, intl } = this.props;
-    const { expired } = this.state;
+    const { revealed, expired } = this.state;
 
     if (!poll) {
       return null;
     }
 
     const timeRemaining = expired ? intl.formatMessage(messages.closed) : <RelativeTimestamp timestamp={poll.get('expires_at')} futureDate />;
-    const showResults   = poll.get('voted') || expired;
+    const showResults   = poll.get('voted') || revealed || expired;
     const disabled      = this.props.disabled || Object.entries(this.state.selected).every(item => !item);
 
     let votesCount = null;
@@ -231,9 +235,10 @@ class Poll extends ImmutablePureComponent {
 
         <div className='poll__footer'>
           {!showResults && <button className='button button-secondary' disabled={disabled || !this.context.identity.signedIn} onClick={this.handleVote}><FormattedMessage id='poll.vote' defaultMessage='Vote' /></button>}
-          {showResults && !this.props.disabled && <span><button className='poll__link' onClick={this.handleRefresh}><FormattedMessage id='poll.refresh' defaultMessage='Refresh' /></button> · </span>}
+          {!showResults && <><button className='poll__link' onClick={this.handleReveal}><FormattedMessage id='poll.reveal' defaultMessage='See results' /></button> · </>}
+          {showResults && !this.props.disabled && <><button className='poll__link' onClick={this.handleRefresh}><FormattedMessage id='poll.refresh' defaultMessage='Refresh' /></button> · </>}
           {votesCount}
-          {poll.get('expires_at') && <span> · {timeRemaining}</span>}
+          {poll.get('expires_at') && <> · {timeRemaining}</>}
         </div>
       </div>
     );

app/javascript/mastodon/components/status_content.jsx

@@ -104,7 +104,7 @@ class StatusContent extends PureComponent {
 
       if (mention) {
         link.addEventListener('click', this.onMentionClick.bind(this, mention), false);
-        link.setAttribute('title', mention.get('acct'));
+        link.setAttribute('title', `@${mention.get('acct')}`);
         link.setAttribute('href', `/@${mention.get('acct')}`);
       } else if (link.textContent[0] === '#' || (link.previousSibling && link.previousSibling.textContent && link.previousSibling.textContent[link.previousSibling.textContent.length - 1] === '#')) {
         link.addEventListener('click', this.onHashtagClick.bind(this, link.text), false);

app/javascript/mastodon/features/account/components/header.jsx

@@ -476,6 +476,7 @@ class Header extends ImmutablePureComponent {
         <Helmet>
           <title>{titleFromAccount(account)}</title>
           <meta name='robots' content={(isLocal && isIndexable) ? 'all' : 'noindex'} />
+          <link rel='canonical' href={account.get('url')} />
         </Helmet>
       </div>
     );

app/javascript/mastodon/features/firehose/index.jsx

@@ -84,7 +84,7 @@ const Firehose = ({ feedType, multiColumn }) => {
     (maxId) => {
       switch(feedType) {
       case 'community':
-        dispatch(expandCommunityTimeline({ onlyMedia }));
+        dispatch(expandCommunityTimeline({ maxId, onlyMedia }));
         break;
       case 'public':
         dispatch(expandPublicTimeline({ maxId, onlyMedia }));
@@ -138,7 +138,8 @@ const Firehose = ({ feedType, multiColumn }) => {
     <DismissableBanner id='public_timeline'>
       <FormattedMessage
         id='dismissable_banner.public_timeline'
-       defaultMessage='These are the most recent public posts from people on this and other servers of the decentralized network that this server knows about.'
+        defaultMessage='These are the most recent public posts from people on the social web that people on {domain} follow.'
+        values={{ domain }}
       />
     </DismissableBanner>
   );
@@ -171,11 +172,11 @@ const Firehose = ({ feedType, multiColumn }) => {
       <div className='scrollable scrollable--flex'>
         <div className='account__section-headline'>
           <NavLink exact to='/public/local'>
-            <FormattedMessage tagName='div' id='firehose.local' defaultMessage='Local' />
+            <FormattedMessage tagName='div' id='firehose.local' defaultMessage='This server' />
           </NavLink>
 
           <NavLink exact to='/public/remote'>
-            <FormattedMessage tagName='div' id='firehose.remote' defaultMessage='Remote' />
+            <FormattedMessage tagName='div' id='firehose.remote' defaultMessage='Other servers' />
           </NavLink>
 
           <NavLink exact to='/public'>

app/javascript/mastodon/features/public_timeline/index.jsx

@@ -8,6 +8,7 @@ import { Helmet } from 'react-helmet';
 import { connect } from 'react-redux';
 
 import DismissableBanner from 'mastodon/components/dismissable_banner';
+import { domain } from 'mastodon/initial_state';
 
 import { addColumn, removeColumn, moveColumn } from '../../actions/columns';
 import { connectPublicStream } from '../../actions/streaming';
@@ -143,7 +144,7 @@ class PublicTimeline extends PureComponent {
         </ColumnHeader>
 
         <StatusListContainer
-          prepend={<DismissableBanner id='public_timeline'><FormattedMessage id='dismissable_banner.public_timeline' defaultMessage='These are the most recent public posts from people on this and other servers of the decentralized network that this server knows about.' /></DismissableBanner>}
+          prepend={<DismissableBanner id='public_timeline'><FormattedMessage id='dismissable_banner.public_timeline' defaultMessage='These are the most recent public posts from people on the social web that people on {domain} follow.' values={{ domain }} /></DismissableBanner>}
           timelineId={`public${onlyRemote ? ':remote' : ''}${onlyMedia ? ':media' : ''}`}
           onLoadMore={this.handleLoadMore}
           trackScroll={!pinned}

app/javascript/mastodon/features/status/index.jsx

@@ -716,6 +716,7 @@ class Status extends ImmutablePureComponent {
         <Helmet>
           <title>{titleFromStatus(intl, status)}</title>
           <meta name='robots' content={(isLocal && isIndexable) ? 'all' : 'noindex'} />
+          <link rel='canonical' href={status.get('url')} />
         </Helmet>
       </Column>
     );

app/javascript/mastodon/locales/en.json

@@ -202,7 +202,7 @@
   "dismissable_banner.explore_links": "These are news stories being shared the most on the social web today. Newer news stories posted by more different people are ranked higher.",
   "dismissable_banner.explore_statuses": "These are posts from across the social web that are gaining traction today. Newer posts with more boosts and favourites are ranked higher.",
   "dismissable_banner.explore_tags": "These are hashtags that are gaining traction on the social web today. Hashtags that are used by more different people are ranked higher.",
-  "dismissable_banner.public_timeline": "These are the most recent public posts from people on this and other servers of the decentralized network that this server knows about.",
+  "dismissable_banner.public_timeline": "These are the most recent public posts from people on the social web that people on {domain} follow.",
   "embed.instructions": "Embed this post on your website by copying the code below.",
   "embed.preview": "Here is what it will look like:",
   "emoji_button.activity": "Activity",
@@ -269,8 +269,8 @@
   "filter_modal.select_filter.title": "Filter this post",
   "filter_modal.title.status": "Filter a post",
   "firehose.all": "All",
-  "firehose.local": "Local",
+  "firehose.local": "This server",
-  "firehose.remote": "Remote",
+  "firehose.remote": "Other servers",
   "follow_request.authorize": "Authorize",
   "follow_request.reject": "Reject",
   "follow_requests.unlocked_explanation": "Even though your account is not locked, the {domain} staff thought you might want to review follow requests from these accounts manually.",
@@ -487,6 +487,7 @@
   "picture_in_picture.restore": "Put it back",
   "poll.closed": "Closed",
   "poll.refresh": "Refresh",
+  "poll.reveal": "See results",
   "poll.total_people": "{count, plural, one {# person} other {# people}}",
   "poll.total_votes": "{count, plural, one {# vote} other {# votes}}",
   "poll.vote": "Vote",

app/javascript/styles/mastodon-light/diff.scss

@@ -5,19 +5,6 @@ html {
   scrollbar-color: $ui-base-color rgba($ui-base-color, 0.25);
 }
 
-// Change the colors of button texts
-.button {
-  color: $white;
-
-  &.button-alternative-2 {
-    color: $white;
-  }
-
-  &.button-tertiary {
-    color: $highlight-text-color;
-  }
-}
-
 .simple_form .button.button-tertiary {
   color: $highlight-text-color;
 }
@@ -436,26 +423,6 @@ html {
   color: $white;
 }
 
-.button.button-tertiary {
-  &:hover,
-  &:focus,
-  &:active {
-    color: $white;
-  }
-}
-
-.button.button-secondary {
-  border-color: $darker-text-color;
-  color: $darker-text-color;
-
-  &:hover,
-  &:focus,
-  &:active {
-    border-color: darken($darker-text-color, 8%);
-    color: darken($darker-text-color, 8%);
-  }
-}
-
 .flash-message.warning {
   color: lighten($gold-star, 16%);
 }

app/javascript/styles/mastodon-light/variables.scss

@@ -7,6 +7,12 @@ $classic-primary-color: #9baec8;
 $classic-secondary-color: #d9e1e8;
 $classic-highlight-color: #6364ff;
 
+$blurple-600: #563acc; // Iris
+$blurple-500: #6364ff; // Brand purple
+$blurple-300: #858afa; // Faded Blue
+$grey-600: #4e4c5a; // Trout
+$grey-100: #dadaf3; // Topaz
+
 // Differences
 $success-green: lighten(#3c754d, 8%);
 
@@ -19,6 +25,13 @@ $ui-primary-color: #9bcbed;
 $ui-secondary-color: $classic-base-color !default;
 $ui-highlight-color: $classic-highlight-color !default;
 
+$ui-button-secondary-color: $grey-600 !default;
+$ui-button-secondary-border-color: $grey-600 !default;
+$ui-button-secondary-focus-color: $white !default;
+
+$ui-button-tertiary-color: $blurple-500 !default;
+$ui-button-tertiary-border-color: $blurple-500 !default;
+
 $primary-text-color: $black !default;
 $darker-text-color: $classic-base-color !default;
 $highlight-text-color: darken($ui-highlight-color, 8%) !default;

app/javascript/styles/mastodon/admin.scss

@@ -128,7 +128,6 @@ $content-width: 840px;
         }
 
         &.selected {
-          background: darken($ui-base-color, 2%);
           border-radius: 4px 0 0;
         }
       }
@@ -146,13 +145,9 @@ $content-width: 840px;
 
       .simple-navigation-active-leaf a {
         color: $primary-text-color;
-        background-color: darken($ui-highlight-color, 2%);
+        background-color: $ui-highlight-color;
         border-bottom: 0;
         border-radius: 0;
-
-        &:hover {
-          background-color: $ui-highlight-color;
-        }
       }
     }
 
@@ -246,12 +241,6 @@ $content-width: 840px;
             font-weight: 700;
             color: $primary-text-color;
             background: $ui-highlight-color;
-
-            &:hover,
-            &:focus,
-            &:active {
-              background: lighten($ui-highlight-color, 4%);
-            }
           }
         }
       }

app/javascript/styles/mastodon/components.scss

@@ -47,11 +47,11 @@
 }
 
 .button {
-  background-color: darken($ui-highlight-color, 2%);
+  background-color: $ui-button-background-color;
   border: 10px none;
   border-radius: 4px;
   box-sizing: border-box;
-  color: $primary-text-color;
+  color: $ui-button-color;
   cursor: pointer;
   display: inline-block;
   font-family: inherit;
@@ -71,14 +71,14 @@
   &:active,
   &:focus,
   &:hover {
-    background-color: $ui-highlight-color;
+    background-color: $ui-button-focus-background-color;
   }
 
   &--destructive {
     &:active,
     &:focus,
     &:hover {
-      background-color: $error-red;
+      background-color: $ui-button-destructive-focus-background-color;
       transition: none;
     }
   }
@@ -108,39 +108,18 @@
     outline: 0 !important;
   }
 
-  &.button-alternative {
-    color: $inverted-text-color;
-    background: $ui-primary-color;
-
-    &:active,
-    &:focus,
-    &:hover {
-      background-color: lighten($ui-primary-color, 4%);
-    }
-  }
-
-  &.button-alternative-2 {
-    background: $ui-base-lighter-color;
-
-    &:active,
-    &:focus,
-    &:hover {
-      background-color: lighten($ui-base-lighter-color, 4%);
-    }
-  }
-
   &.button-secondary {
-    color: $darker-text-color;
+    color: $ui-button-secondary-color;
     background: transparent;
     padding: 6px 17px;
-    border: 1px solid lighten($ui-base-color, 12%);
+    border: 1px solid $ui-button-secondary-border-color;
 
     &:active,
     &:focus,
     &:hover {
-      background: lighten($ui-base-color, 4%);
+      border-color: $ui-button-secondary-focus-background-color;
-      border-color: lighten($ui-base-color, 16%);
+      color: $ui-button-secondary-focus-color;
-      color: lighten($darker-text-color, 4%);
+      background-color: $ui-button-secondary-focus-background-color;
       text-decoration: none;
     }
 
@@ -152,14 +131,14 @@
   &.button-tertiary {
     background: transparent;
     padding: 6px 17px;
-    color: $highlight-text-color;
+    color: $ui-button-tertiary-color;
-    border: 1px solid $highlight-text-color;
+    border: 1px solid $ui-button-tertiary-border-color;
 
     &:active,
     &:focus,
     &:hover {
-      background: $ui-highlight-color;
+      background-color: $ui-button-tertiary-focus-background-color;
-      color: $primary-text-color;
+      color: $ui-button-tertiary-focus-color;
       border: 0;
       padding: 7px 18px;
     }
@@ -1148,6 +1127,8 @@ body > [data-popper-placement] {
   }
 
   &--in-thread {
+    $thread-margin: 46px + 10px;
+
     border-bottom: 0;
 
     .status__content,
@@ -1158,8 +1139,12 @@ body > [data-popper-placement] {
     .attachment-list,
     .picture-in-picture-placeholder,
     .status-card {
-      margin-inline-start: 46px + 10px;
+      margin-inline-start: $thread-margin;
-      width: calc(100% - (46px + 10px));
+      width: calc(100% - ($thread-margin));
+    }
+
+    .status__content__read-more-button {
+      margin-inline-start: $thread-margin;
     }
   }
 
@@ -5810,15 +5795,15 @@ a.status-card.compact:hover {
   }
 
   .button.button-secondary {
-    border-color: $inverted-text-color;
+    border-color: $ui-button-secondary-border-color;
-    color: $inverted-text-color;
+    color: $ui-button-secondary-color;
     flex: 0 0 auto;
 
     &:hover,
     &:focus,
     &:active {
-      border-color: lighten($inverted-text-color, 15%);
+      border-color: $ui-button-secondary-focus-background-color;
-      color: lighten($inverted-text-color, 15%);
+      color: $ui-button-secondary-focus-color;
     }
   }
 

app/javascript/styles/mastodon/dashboard.scss

@@ -81,7 +81,7 @@
     display: flex;
     align-items: baseline;
     border-radius: 4px;
-    background: darken($ui-highlight-color, 2%);
+    background: $ui-button-background-color;
     color: $primary-text-color;
     transition: all 100ms ease-in;
     font-size: 14px;
@@ -94,7 +94,7 @@
     &:active,
     &:focus,
     &:hover {
-      background-color: $ui-highlight-color;
+      background-color: $ui-button-focus-background-color;
       transition: all 200ms ease-out;
     }
 

app/javascript/styles/mastodon/forms.scss

@@ -511,8 +511,8 @@ code {
     width: 100%;
     border: 0;
     border-radius: 4px;
-    background: darken($ui-highlight-color, 2%);
+    background: $ui-button-background-color;
-    color: $primary-text-color;
+    color: $ui-button-color;
     font-size: 18px;
     line-height: inherit;
     height: auto;
@@ -534,7 +534,7 @@ code {
     &:active,
     &:focus,
     &:hover {
-      background-color: $ui-highlight-color;
+      background-color: $ui-button-focus-background-color;
     }
 
     &:disabled:hover {
@@ -542,15 +542,12 @@ code {
     }
 
     &.negative {
-      background: $error-value-color;
+      background: $ui-button-destructive-background-color;
-
-      &:hover {
-        background-color: lighten($error-value-color, 5%);
-      }
 
+      &:hover,
       &:active,
       &:focus {
-        background-color: darken($error-value-color, 5%);
+        background-color: $ui-button-destructive-focus-background-color;
       }
     }
   }

app/javascript/styles/mastodon/variables.scss

@@ -1,8 +1,16 @@
 // Commonly used web colors
 $black: #000000; // Black
 $white: #ffffff; // White
+$red-600: #b7253d !default; // Deep Carmine
+$red-500: #df405a !default; // Cerise
+$blurple-600: #563acc; // Iris
+$blurple-500: #6364ff; // Brand purple
+$blurple-300: #858afa; // Faded Blue
+$grey-600: #4e4c5a; // Trout
+$grey-100: #dadaf3; // Topaz
+
 $success-green: #79bd9a !default; // Padua
-$error-red: #df405a !default; // Cerise
+$error-red: $red-500 !default; // Cerise
 $warning-red: #ff5050 !default; // Sunset Orange
 $gold-star: #ca8f04 !default; // Dark Goldenrod
 
@@ -31,6 +39,22 @@ $ui-base-lighter-color: lighten(
 $ui-primary-color: $classic-primary-color !default; // Lighter
 $ui-secondary-color: $classic-secondary-color !default; // Lightest
 $ui-highlight-color: $classic-highlight-color !default;
+$ui-button-color: $white !default;
+$ui-button-background-color: $blurple-500 !default;
+$ui-button-focus-background-color: $blurple-600 !default;
+
+$ui-button-secondary-color: $grey-100 !default;
+$ui-button-secondary-border-color: $grey-100 !default;
+$ui-button-secondary-focus-background-color: $grey-600 !default;
+$ui-button-secondary-focus-color: $white !default;
+
+$ui-button-tertiary-color: $blurple-300 !default;
+$ui-button-tertiary-border-color: $blurple-300 !default;
+$ui-button-tertiary-focus-background-color: $blurple-600 !default;
+$ui-button-tertiary-focus-color: $white !default;
+
+$ui-button-destructive-background-color: $red-500 !default;
+$ui-button-destructive-focus-background-color: $red-600 !default;
 
 // Variables for texts
 $primary-text-color: $white !default;
@@ -39,6 +63,7 @@ $dark-text-color: $ui-base-lighter-color !default;
 $secondary-text-color: $ui-secondary-color !default;
 $highlight-text-color: lighten($ui-highlight-color, 8%) !default;
 $action-button-color: $ui-base-lighter-color !default;
+$action-button-focus-color: lighten($ui-base-lighter-color, 4%) !default;
 $passive-text-color: $gold-star !default;
 $active-passive-text-color: $success-green !default;
 

app/lib/request.rb

@@ -7,11 +7,48 @@ require 'resolv'
 # Monkey-patch the HTTP.rb timeout class to avoid using a timeout block
 # around the Socket#open method, since we use our own timeout blocks inside
 # that method
+#
+# Also changes how the read timeout behaves so that it is cumulative (closer
+# to HTTP::Timeout::Global, but still having distinct timeouts for other
+# operation types)
 class HTTP::Timeout::PerOperation
   def connect(socket_class, host, port, nodelay = false)
     @socket = socket_class.open(host, port)
     @socket.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1) if nodelay
   end
+
+  # Reset deadline when the connection is re-used for different requests
+  def reset_counter
+    @deadline = nil
+  end
+
+  # Read data from the socket
+  def readpartial(size, buffer = nil)
+    @deadline ||= Process.clock_gettime(Process::CLOCK_MONOTONIC) + @read_timeout
+
+    timeout = false
+    loop do
+      result = @socket.read_nonblock(size, buffer, exception: false)
+
+      return :eof if result.nil?
+
+      remaining_time = @deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      raise HTTP::TimeoutError, "Read timed out after #{@read_timeout} seconds" if timeout || remaining_time <= 0
+      return result if result != :wait_readable
+
+      # marking the socket for timeout. Why is this not being raised immediately?
+      # it seems there is some race-condition on the network level between calling
+      # #read_nonblock and #wait_readable, in which #read_nonblock signalizes waiting
+      # for reads, and when waiting for x seconds, it returns nil suddenly without completing
+      # the x seconds. In a normal case this would be a timeout on wait/read, but it can
+      # also mean that the socket has been closed by the server. Therefore we "mark" the
+      # socket for timeout and try to read more bytes. If it returns :eof, it's all good, no
+      # timeout. Else, the first timeout was a proper timeout.
+      # This hack has to be done because io/wait#wait_readable doesn't provide a value for when
+      # the socket is closed by the server, and HTTP::Parser doesn't provide the limit for the chunks.
+      timeout = true unless @socket.to_io.wait_readable(remaining_time)
+    end
+  end
 end
 
 class Request

app/lib/scope_parser.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 class ScopeParser < Parslet::Parser
-  rule(:term)      { match('[a-z]').repeat(1).as(:term) }
+  rule(:term)      { match('[a-z_]').repeat(1).as(:term) }
   rule(:colon)     { str(':') }
   rule(:access)    { (str('write') | str('read')).as(:access) }
   rule(:namespace) { str('admin').as(:namespace) }

app/lib/text_formatter.rb

@@ -48,6 +48,26 @@ class TextFormatter
     html.html_safe # rubocop:disable Rails/OutputSafety
   end
 
+  class << self
+    include ERB::Util
+
+    def shortened_link(url, rel_me: false)
+      url = Addressable::URI.parse(url).to_s
+      rel = rel_me ? (DEFAULT_REL + %w(me)) : DEFAULT_REL
+
+      prefix      = url.match(URL_PREFIX_REGEX).to_s
+      display_url = url[prefix.length, 30]
+      suffix      = url[prefix.length + 30..-1]
+      cutoff      = url[prefix.length..-1].length > 30
+
+      <<~HTML.squish
+        <a href="#{h(url)}" target="_blank" rel="#{rel.join(' ')}" translate="no"><span class="invisible">#{h(prefix)}</span><span class="#{cutoff ? 'ellipsis' : ''}">#{h(display_url)}</span><span class="invisible">#{h(suffix)}</span></a>
+      HTML
+    rescue Addressable::URI::InvalidURIError, IDN::Idna::IdnaError
+      h(url)
+    end
+  end
+
   private
 
   def rewrite
@@ -70,19 +90,7 @@ class TextFormatter
   end
 
   def link_to_url(entity)
-    url = Addressable::URI.parse(entity[:url]).to_s
+    TextFormatter.shortened_link(entity[:url], rel_me: with_rel_me?)
-    rel = with_rel_me? ? (DEFAULT_REL + %w(me)) : DEFAULT_REL
-
-    prefix      = url.match(URL_PREFIX_REGEX).to_s
-    display_url = url[prefix.length, 30]
-    suffix      = url[prefix.length + 30..-1]
-    cutoff      = url[prefix.length..-1].length > 30
-
-    <<~HTML.squish
-      <a href="#{h(url)}" target="_blank" rel="#{rel.join(' ')}" translate="no"><span class="invisible">#{h(prefix)}</span><span class="#{cutoff ? 'ellipsis' : ''}">#{h(display_url)}</span><span class="invisible">#{h(suffix)}</span></a>
-    HTML
-  rescue Addressable::URI::InvalidURIError, IDN::Idna::IdnaError
-    h(entity[:url])
   end
 
   def link_to_hashtag(entity)

app/models/concerns/attachmentable.rb

@@ -22,15 +22,14 @@ module Attachmentable
 
   included do
     def self.has_attached_file(name, options = {}) # rubocop:disable Naming/PredicateName
-      options = { validate_media_type: false }.merge(options)
       super(name, options)
-      send(:"before_#{name}_post_process") do
+
+      send(:"before_#{name}_validate") do
         attachment = send(name)
         check_image_dimension(attachment)
         set_file_content_type(attachment)
         obfuscate_file_name(attachment)
         set_file_extension(attachment)
-        Paperclip::Validators::MediaTypeSpoofDetectionValidator.new(attributes: [name]).validate(self)
       end
     end
   end

app/serializers/rest/preview_card_serializer.rb

@@ -11,4 +11,8 @@ class REST::PreviewCardSerializer < ActiveModel::Serializer
   def image
     object.image? ? full_asset_url(object.image.url(:original)) : nil
   end
+
+  def html
+    Sanitize.fragment(object.html, Sanitize::Config::MASTODON_OEMBED)
+  end
 end

app/services/search_service.rb

@@ -8,6 +8,7 @@ class SearchService < BaseService
     @limit     = limit.to_i
     @offset    = options[:type].blank? ? 0 : options[:offset].to_i
     @resolve   = options[:resolve] || false
+    @following = options[:following] || false
 
     default_results.tap do |results|
       next if @query.blank? || @limit.zero?
@@ -31,7 +32,8 @@ class SearchService < BaseService
       limit: @limit,
       resolve: @resolve,
       offset: @offset,
-      use_searchable_text: true
+      use_searchable_text: true,
+      following: @following
     )
   end
 

app/workers/account_deletion_worker.rb

@@ -6,9 +6,12 @@ class AccountDeletionWorker
   sidekiq_options queue: 'pull', lock: :until_executed
 
   def perform(account_id, options = {})
+    account = Account.find(account_id)
+    return unless account.suspended?
+
     reserve_username = options.with_indifferent_access.fetch(:reserve_username, true)
     skip_activitypub = options.with_indifferent_access.fetch(:skip_activitypub, false)
-    DeleteAccountService.new.call(Account.find(account_id), reserve_username: reserve_username, skip_activitypub: skip_activitypub, reserve_email: false)
+    DeleteAccountService.new.call(account, reserve_username: reserve_username, skip_activitypub: skip_activitypub, reserve_email: false)
   rescue ActiveRecord::RecordNotFound
     true
   end

config/application.rb

@@ -28,6 +28,7 @@ require_relative '../lib/paperclip/url_generator_extensions'
 require_relative '../lib/paperclip/attachment_extensions'
 require_relative '../lib/paperclip/lazy_thumbnail'
 require_relative '../lib/paperclip/gif_transcoder'
+require_relative '../lib/paperclip/media_type_spoof_detector_extensions'
 require_relative '../lib/paperclip/transcoder'
 require_relative '../lib/paperclip/type_corrector'
 require_relative '../lib/paperclip/response_with_limit_adapter'

config/imagemagick/policy.xml

@@ -0,0 +1,27 @@
+<policymap>
+  <!-- Set some basic system resource limits -->
+  <policy domain="resource" name="time" value="60" />
+
+  <policy domain="module" rights="none" pattern="URL" />
+
+  <policy domain="filter" rights="none" pattern="*" />
+
+  <!--
+    Ideally, we would restrict ImageMagick to only accessing its own
+    disk-backed pixel cache as well as Mastodon-created Tempfiles.
+
+    However, those paths depend on the operating system and environment
+    variables, so they can only be known at runtime.
+
+    Furthermore, those paths are not necessarily shared across Mastodon
+    processes, so even creating a policy.xml at runtime is impractical.
+
+    For the time being, only disable indirect reads.
+  -->
+  <policy domain="path" rights="none" pattern="@*" />
+
+  <!-- Disallow any coder by default, and only enable ones required by Mastodon -->
+  <policy domain="coder" rights="none" pattern="*" />
+  <policy domain="coder" rights="read | write" pattern="{PNG,JPEG,GIF,HEIC,WEBP}" />
+  <policy domain="coder" rights="write" pattern="{HISTOGRAM,RGB,INFO}" />
+</policymap>

config/initializers/paperclip.rb

@@ -153,3 +153,10 @@ unless defined?(Seahorse)
     end
   end
 end
+
+# Set our ImageMagick security policy, but allow admins to override it
+ENV['MAGICK_CONFIGURE_PATH'] = begin
+  imagemagick_config_paths = ENV.fetch('MAGICK_CONFIGURE_PATH', '').split(File::PATH_SEPARATOR)
+  imagemagick_config_paths << Rails.root.join('config', 'imagemagick').expand_path.to_s
+  imagemagick_config_paths.join(File::PATH_SEPARATOR)
+end

config/routes.rb

@@ -13,6 +13,7 @@ Rails.application.routes.draw do
     /home
     /public
     /public/local
+    /public/remote
     /conversations
     /lists/(*any)
     /notifications

db/migrate/20230702131023_add_superapp_index_to_applications.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+class AddSuperappIndexToApplications < ActiveRecord::Migration[6.1]
+  disable_ddl_transaction!
+
+  def change
+    add_index :oauth_applications, :superapp, where: 'superapp = true', algorithm: :concurrently
+  end
+end

db/migrate/20230702151753_add_index_user_on_unconfirmed_email.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+class AddIndexUserOnUnconfirmedEmail < ActiveRecord::Migration[6.1]
+  disable_ddl_transaction!
+
+  def change
+    add_index :users, :unconfirmed_email, where: 'unconfirmed_email IS NOT NULL', algorithm: :concurrently
+  end
+end

db/schema.rb

@@ -10,7 +10,7 @@
 #
 # It's strongly recommended that you check this file into your version control system.
 
-ActiveRecord::Schema.define(version: 2023_06_30_145300) do
+ActiveRecord::Schema.define(version: 2023_07_02_151753) do
 
   # These are extensions that must be enabled in order to support this database
   enable_extension "plpgsql"
@@ -700,6 +700,7 @@ ActiveRecord::Schema.define(version: 2023_06_30_145300) do
     t.bigint "owner_id"
     t.boolean "confidential", default: true, null: false
     t.index ["owner_id", "owner_type"], name: "index_oauth_applications_on_owner_id_and_owner_type"
+    t.index ["superapp"], name: "index_oauth_applications_on_superapp", where: "(superapp = true)"
     t.index ["uid"], name: "index_oauth_applications_on_uid", unique: true
   end
 
@@ -1101,6 +1102,7 @@ ActiveRecord::Schema.define(version: 2023_06_30_145300) do
     t.index ["email"], name: "index_users_on_email", unique: true
     t.index ["reset_password_token"], name: "index_users_on_reset_password_token", unique: true, opclass: :text_pattern_ops, where: "(reset_password_token IS NOT NULL)"
     t.index ["role_id"], name: "index_users_on_role_id", where: "(role_id IS NOT NULL)"
+    t.index ["unconfirmed_email"], name: "index_users_on_unconfirmed_email", where: "(unconfirmed_email IS NOT NULL)"
   end
 
   create_table "web_push_subscriptions", force: :cascade do |t|

dist/nginx.conf

@@ -109,6 +109,8 @@ server {
   location ~ ^/system/ {
     add_header Cache-Control "public, max-age=2419200, immutable";
     add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
+    add_header X-Content-Type-Options nosniff;
+    add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
     try_files $uri =404;
   }
 

lib/mastodon/version.rb

@@ -13,7 +13,7 @@ module Mastodon
     end
 
     def patch
-      2
+      3
     end
 
     def flags

lib/paperclip/media_type_spoof_detector_extensions.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module Paperclip
+  module MediaTypeSpoofDetectorExtensions
+    def calculated_content_type
+      return @calculated_content_type if defined?(@calculated_content_type)
+
+      @calculated_content_type = type_from_file_command.chomp
+
+      # The `file` command fails to recognize some MP3 files as such
+      @calculated_content_type = type_from_marcel if @calculated_content_type == 'application/octet-stream' && type_from_marcel == 'audio/mpeg'
+      @calculated_content_type
+    end
+
+    def type_from_marcel
+      @type_from_marcel ||= Marcel::MimeType.for Pathname.new(@file.path),
+                                                 name: @file.path
+    end
+  end
+end
+
+Paperclip::MediaTypeSpoofDetector.prepend(Paperclip::MediaTypeSpoofDetectorExtensions)

lib/paperclip/transcoder.rb

@@ -19,10 +19,7 @@ module Paperclip
     def make
       metadata = VideoMetadataExtractor.new(@file.path)
 
-      unless metadata.valid?
+      raise Paperclip::Error, "Error while transcoding #{@file.path}: unsupported file" unless metadata.valid?
-        Paperclip.log("Unsupported file #{@file.path}")
-        return File.open(@file.path)
-      end
 
       update_attachment_type(metadata)
       update_options_from_metadata(metadata)

lib/public_file_server_middleware.rb

@@ -32,6 +32,11 @@ class PublicFileServerMiddleware
       end
     end
 
+    # Override the default CSP header set by the CSP middleware
+    headers['Content-Security-Policy'] = "default-src 'none'; form-action 'none'" if request_path.start_with?(paperclip_root_url)
+
+    headers['X-Content-Type-Options'] = 'nosniff'
+
     [status, headers, response]
   end
 

lib/sanitize_ext/sanitize_config.rb

@@ -107,26 +107,26 @@ class Sanitize
       ]
     )
 
-    MASTODON_OEMBED ||= freeze_config merge(
+    MASTODON_OEMBED ||= freeze_config(
-      RELAXED,
+      elements: %w(audio embed iframe source video),
-      elements: RELAXED[:elements] + %w(audio embed iframe source video),
 
-      attributes: merge(
+      attributes: {
-        RELAXED[:attributes],
         'audio' => %w(controls),
         'embed' => %w(height src type width),
         'iframe' => %w(allowfullscreen frameborder height scrolling src width),
         'source' => %w(src type),
         'video' => %w(controls height loop width),
-        'div' => [:data]
+      },
-      ),
 
-      protocols: merge(
+      protocols: {
-        RELAXED[:protocols],
         'embed' => { 'src' => HTTP_PROTOCOLS },
         'iframe' => { 'src' => HTTP_PROTOCOLS },
-        'source' => { 'src' => HTTP_PROTOCOLS }
+        'source' => { 'src' => HTTP_PROTOCOLS },
-      )
+      },
+
+      add_attributes: {
+        'iframe' => { 'sandbox' => 'allow-scripts allow-same-origin allow-popups allow-popups-to-escape-sandbox allow-forms' },
+      }
     )
 
     LINK_REL_TRANSFORMER = lambda do |env|

spec/controllers/api/v2/search_controller_spec.rb

@@ -14,13 +14,40 @@ RSpec.describe Api::V2::SearchController do
     end
 
     describe 'GET #index' do
-      before do
+      let!(:bob)   { Fabricate(:account, username: 'bob_test') }
-        get :index, params: { q: 'test' }
+      let!(:ana)   { Fabricate(:account, username: 'ana_test') }
-      end
+      let!(:tom)   { Fabricate(:account, username: 'tom_test') }
+      let(:params) { { q: 'test' } }
 
       it 'returns http success' do
+        get :index, params: params
+
         expect(response).to have_http_status(200)
       end
+
+      context 'when searching accounts' do
+        let(:params) { { q: 'test', type: 'accounts' } }
+
+        it 'returns all matching accounts' do
+          get :index, params: params
+
+          expect(body_as_json[:accounts].pluck(:id)).to contain_exactly(bob.id.to_s, ana.id.to_s, tom.id.to_s)
+        end
+
+        context 'with following=true' do
+          let(:params) { { q: 'test', type: 'accounts', following: 'true' } }
+
+          before do
+            user.account.follow!(ana)
+          end
+
+          it 'returns only the followed accounts' do
+            get :index, params: params
+
+            expect(body_as_json[:accounts].pluck(:id)).to contain_exactly(ana.id.to_s)
+          end
+        end
+      end
     end
   end
 

spec/models/media_attachment_spec.rb

@@ -152,6 +152,26 @@ RSpec.describe MediaAttachment, paperclip_processing: true do
     end
   end
 
+  describe 'mp3 with large cover art' do
+    let(:media) { described_class.create(account: Fabricate(:account), file: attachment_fixture('boop.mp3')) }
+
+    it 'detects it as an audio file' do
+      expect(media.type).to eq 'audio'
+    end
+
+    it 'sets meta for the duration' do
+      expect(media.file.meta['original']['duration']).to be_within(0.05).of(0.235102)
+    end
+
+    it 'extracts thumbnail' do
+      expect(media.thumbnail.present?).to be true
+    end
+
+    it 'gives the file a random name' do
+      expect(media.file_file_name).to_not eq 'boop.mp3'
+    end
+  end
+
   describe 'jpeg' do
     let(:media) { described_class.create(account: Fabricate(:account), file: attachment_fixture('attachment.jpg')) }
 

spec/services/search_service_spec.rb

@@ -68,7 +68,7 @@ describe SearchService, type: :service do
           allow(AccountSearchService).to receive(:new).and_return(service)
 
           results = subject.call(query, nil, 10)
-          expect(service).to have_received(:call).with(query, nil, limit: 10, offset: 0, resolve: false, use_searchable_text: true)
+          expect(service).to have_received(:call).with(query, nil, limit: 10, offset: 0, resolve: false, use_searchable_text: true, following: false)
           expect(results).to eq empty_results.merge(accounts: [account])
         end
       end