mastodon/app/models/admin/import.rb

65 lines
1.6 KiB
Ruby
Raw Normal View History

Support for import/export of instance-level domain blocks/allows for 4.x w/ additional fixes (#20597) * Allow import/export of instance-level domain blocks/allows (#1754) * Allow import/export of instance-level domain blocks/allows. Fixes #15095 * Pacify circleci * Address simple code review feedback * Add headers to exported CSV * Extract common import/export functionality to AdminExportControllerConcern * Add additional fields to instance-blocked domain export * Address review feedback * Split instance domain block/allow import/export into separate pages/controllers * Address code review feedback * Pacify DeepSource * Work around Paperclip::HasAttachmentFile for Rails 6 * Fix deprecated API warning in export tests * Remove after_commit workaround (cherry picked from commit b8319d057824bc4bcdbac2cad16f97e264844168) * Add confirmation page when importing blocked domains (#1773) * Move glitch-soc-specific strings to glitch-soc-specific locale files * Add confirmation page when importing blocked domains (cherry picked from commit 73a2e3326fe25d52d7067d946be509aeb0eea5c2) * Fix authorization check in domain blocks controller (cherry picked from commit 655bac2c3aed67eed868d32d1a5b02df4c46e6b4) * Fix error strings for domain blocks and email-domain blocks Corrected issue with non-error message used for Mastodon:NotPermittedError in Domain Blocks Corrected issue Domain Blocks using the Email Domain Blocks message on ActionContoller::ParameterMissing Corrected issue with Email Domain Blocks using the not_permitted string from "custom emojii's" * Ran i18n-tasks normalize to address test failure * Removed unused admin.export_domain_blocks.not_permitted string Removing unused string as indicated by Check i18n * Fix tests (cherry picked from commit 1ba41b34c1dfc872f0f19c146a38b0b21e4ee3f7) * Fix domain block export not exporting blocks with only media rejection (cherry picked from commit 1686374b2f2a380de63b616be56be7db918b35f0) * Fix various issues with domain block import - stop using Paperclip for processing domain allow/block imports - stop leaving temporary files - better error handling - assume CSV files are UTF-8-encoded (cherry picked from commit cad824d8f501b95377e4f0a957e5a00d517a1902) Co-authored-by: Levi Bard <taktaktaktaktaktaktaktaktaktak@gmail.com> Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-11-17 10:05:09 +00:00
# frozen_string_literal: true
require 'csv'
Support for import/export of instance-level domain blocks/allows for 4.x w/ additional fixes (#20597) * Allow import/export of instance-level domain blocks/allows (#1754) * Allow import/export of instance-level domain blocks/allows. Fixes #15095 * Pacify circleci * Address simple code review feedback * Add headers to exported CSV * Extract common import/export functionality to AdminExportControllerConcern * Add additional fields to instance-blocked domain export * Address review feedback * Split instance domain block/allow import/export into separate pages/controllers * Address code review feedback * Pacify DeepSource * Work around Paperclip::HasAttachmentFile for Rails 6 * Fix deprecated API warning in export tests * Remove after_commit workaround (cherry picked from commit b8319d057824bc4bcdbac2cad16f97e264844168) * Add confirmation page when importing blocked domains (#1773) * Move glitch-soc-specific strings to glitch-soc-specific locale files * Add confirmation page when importing blocked domains (cherry picked from commit 73a2e3326fe25d52d7067d946be509aeb0eea5c2) * Fix authorization check in domain blocks controller (cherry picked from commit 655bac2c3aed67eed868d32d1a5b02df4c46e6b4) * Fix error strings for domain blocks and email-domain blocks Corrected issue with non-error message used for Mastodon:NotPermittedError in Domain Blocks Corrected issue Domain Blocks using the Email Domain Blocks message on ActionContoller::ParameterMissing Corrected issue with Email Domain Blocks using the not_permitted string from "custom emojii's" * Ran i18n-tasks normalize to address test failure * Removed unused admin.export_domain_blocks.not_permitted string Removing unused string as indicated by Check i18n * Fix tests (cherry picked from commit 1ba41b34c1dfc872f0f19c146a38b0b21e4ee3f7) * Fix domain block export not exporting blocks with only media rejection (cherry picked from commit 1686374b2f2a380de63b616be56be7db918b35f0) * Fix various issues with domain block import - stop using Paperclip for processing domain allow/block imports - stop leaving temporary files - better error handling - assume CSV files are UTF-8-encoded (cherry picked from commit cad824d8f501b95377e4f0a957e5a00d517a1902) Co-authored-by: Levi Bard <taktaktaktaktaktaktaktaktaktak@gmail.com> Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-11-17 10:05:09 +00:00
# A non-activerecord helper class for csv upload
class Admin::Import
include ActiveModel::Model
ROWS_PROCESSING_LIMIT = 20_000
attr_accessor :data
validates :data, presence: true
validate :validate_data
def data_file_name
data.original_filename
end
def csv_rows
csv_data.rewind
csv_data.take(ROWS_PROCESSING_LIMIT + 1)
end
Support for import/export of instance-level domain blocks/allows for 4.x w/ additional fixes (#20597) * Allow import/export of instance-level domain blocks/allows (#1754) * Allow import/export of instance-level domain blocks/allows. Fixes #15095 * Pacify circleci * Address simple code review feedback * Add headers to exported CSV * Extract common import/export functionality to AdminExportControllerConcern * Add additional fields to instance-blocked domain export * Address review feedback * Split instance domain block/allow import/export into separate pages/controllers * Address code review feedback * Pacify DeepSource * Work around Paperclip::HasAttachmentFile for Rails 6 * Fix deprecated API warning in export tests * Remove after_commit workaround (cherry picked from commit b8319d057824bc4bcdbac2cad16f97e264844168) * Add confirmation page when importing blocked domains (#1773) * Move glitch-soc-specific strings to glitch-soc-specific locale files * Add confirmation page when importing blocked domains (cherry picked from commit 73a2e3326fe25d52d7067d946be509aeb0eea5c2) * Fix authorization check in domain blocks controller (cherry picked from commit 655bac2c3aed67eed868d32d1a5b02df4c46e6b4) * Fix error strings for domain blocks and email-domain blocks Corrected issue with non-error message used for Mastodon:NotPermittedError in Domain Blocks Corrected issue Domain Blocks using the Email Domain Blocks message on ActionContoller::ParameterMissing Corrected issue with Email Domain Blocks using the not_permitted string from "custom emojii's" * Ran i18n-tasks normalize to address test failure * Removed unused admin.export_domain_blocks.not_permitted string Removing unused string as indicated by Check i18n * Fix tests (cherry picked from commit 1ba41b34c1dfc872f0f19c146a38b0b21e4ee3f7) * Fix domain block export not exporting blocks with only media rejection (cherry picked from commit 1686374b2f2a380de63b616be56be7db918b35f0) * Fix various issues with domain block import - stop using Paperclip for processing domain allow/block imports - stop leaving temporary files - better error handling - assume CSV files are UTF-8-encoded (cherry picked from commit cad824d8f501b95377e4f0a957e5a00d517a1902) Co-authored-by: Levi Bard <taktaktaktaktaktaktaktaktaktak@gmail.com> Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-11-17 10:05:09 +00:00
private
def csv_data
return @csv_data if defined?(@csv_data)
csv_converter = lambda do |field, field_info|
case field_info.header
when '#domain', '#public_comment'
field&.strip
when '#severity'
field&.strip&.to_sym
when '#reject_media', '#reject_reports', '#obfuscate'
ActiveModel::Type::Boolean.new.cast(field)
else
field
end
end
@csv_data = CSV.open(data.path, encoding: 'UTF-8', skip_blanks: true, headers: true, converters: csv_converter)
@csv_data.take(1) # Ensure the headers are read
@csv_data = CSV.open(data.path, encoding: 'UTF-8', skip_blanks: true, headers: ['#domain'], converters: csv_converter) unless @csv_data.headers&.first == '#domain'
@csv_data
end
Support for import/export of instance-level domain blocks/allows for 4.x w/ additional fixes (#20597) * Allow import/export of instance-level domain blocks/allows (#1754) * Allow import/export of instance-level domain blocks/allows. Fixes #15095 * Pacify circleci * Address simple code review feedback * Add headers to exported CSV * Extract common import/export functionality to AdminExportControllerConcern * Add additional fields to instance-blocked domain export * Address review feedback * Split instance domain block/allow import/export into separate pages/controllers * Address code review feedback * Pacify DeepSource * Work around Paperclip::HasAttachmentFile for Rails 6 * Fix deprecated API warning in export tests * Remove after_commit workaround (cherry picked from commit b8319d057824bc4bcdbac2cad16f97e264844168) * Add confirmation page when importing blocked domains (#1773) * Move glitch-soc-specific strings to glitch-soc-specific locale files * Add confirmation page when importing blocked domains (cherry picked from commit 73a2e3326fe25d52d7067d946be509aeb0eea5c2) * Fix authorization check in domain blocks controller (cherry picked from commit 655bac2c3aed67eed868d32d1a5b02df4c46e6b4) * Fix error strings for domain blocks and email-domain blocks Corrected issue with non-error message used for Mastodon:NotPermittedError in Domain Blocks Corrected issue Domain Blocks using the Email Domain Blocks message on ActionContoller::ParameterMissing Corrected issue with Email Domain Blocks using the not_permitted string from "custom emojii's" * Ran i18n-tasks normalize to address test failure * Removed unused admin.export_domain_blocks.not_permitted string Removing unused string as indicated by Check i18n * Fix tests (cherry picked from commit 1ba41b34c1dfc872f0f19c146a38b0b21e4ee3f7) * Fix domain block export not exporting blocks with only media rejection (cherry picked from commit 1686374b2f2a380de63b616be56be7db918b35f0) * Fix various issues with domain block import - stop using Paperclip for processing domain allow/block imports - stop leaving temporary files - better error handling - assume CSV files are UTF-8-encoded (cherry picked from commit cad824d8f501b95377e4f0a957e5a00d517a1902) Co-authored-by: Levi Bard <taktaktaktaktaktaktaktaktaktak@gmail.com> Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-11-17 10:05:09 +00:00
def csv_row_count
return @csv_row_count if defined?(@csv_row_count)
Support for import/export of instance-level domain blocks/allows for 4.x w/ additional fixes (#20597) * Allow import/export of instance-level domain blocks/allows (#1754) * Allow import/export of instance-level domain blocks/allows. Fixes #15095 * Pacify circleci * Address simple code review feedback * Add headers to exported CSV * Extract common import/export functionality to AdminExportControllerConcern * Add additional fields to instance-blocked domain export * Address review feedback * Split instance domain block/allow import/export into separate pages/controllers * Address code review feedback * Pacify DeepSource * Work around Paperclip::HasAttachmentFile for Rails 6 * Fix deprecated API warning in export tests * Remove after_commit workaround (cherry picked from commit b8319d057824bc4bcdbac2cad16f97e264844168) * Add confirmation page when importing blocked domains (#1773) * Move glitch-soc-specific strings to glitch-soc-specific locale files * Add confirmation page when importing blocked domains (cherry picked from commit 73a2e3326fe25d52d7067d946be509aeb0eea5c2) * Fix authorization check in domain blocks controller (cherry picked from commit 655bac2c3aed67eed868d32d1a5b02df4c46e6b4) * Fix error strings for domain blocks and email-domain blocks Corrected issue with non-error message used for Mastodon:NotPermittedError in Domain Blocks Corrected issue Domain Blocks using the Email Domain Blocks message on ActionContoller::ParameterMissing Corrected issue with Email Domain Blocks using the not_permitted string from "custom emojii's" * Ran i18n-tasks normalize to address test failure * Removed unused admin.export_domain_blocks.not_permitted string Removing unused string as indicated by Check i18n * Fix tests (cherry picked from commit 1ba41b34c1dfc872f0f19c146a38b0b21e4ee3f7) * Fix domain block export not exporting blocks with only media rejection (cherry picked from commit 1686374b2f2a380de63b616be56be7db918b35f0) * Fix various issues with domain block import - stop using Paperclip for processing domain allow/block imports - stop leaving temporary files - better error handling - assume CSV files are UTF-8-encoded (cherry picked from commit cad824d8f501b95377e4f0a957e5a00d517a1902) Co-authored-by: Levi Bard <taktaktaktaktaktaktaktaktaktak@gmail.com> Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-11-17 10:05:09 +00:00
csv_data.rewind
@csv_row_count = csv_data.take(ROWS_PROCESSING_LIMIT + 2).count
end
Support for import/export of instance-level domain blocks/allows for 4.x w/ additional fixes (#20597) * Allow import/export of instance-level domain blocks/allows (#1754) * Allow import/export of instance-level domain blocks/allows. Fixes #15095 * Pacify circleci * Address simple code review feedback * Add headers to exported CSV * Extract common import/export functionality to AdminExportControllerConcern * Add additional fields to instance-blocked domain export * Address review feedback * Split instance domain block/allow import/export into separate pages/controllers * Address code review feedback * Pacify DeepSource * Work around Paperclip::HasAttachmentFile for Rails 6 * Fix deprecated API warning in export tests * Remove after_commit workaround (cherry picked from commit b8319d057824bc4bcdbac2cad16f97e264844168) * Add confirmation page when importing blocked domains (#1773) * Move glitch-soc-specific strings to glitch-soc-specific locale files * Add confirmation page when importing blocked domains (cherry picked from commit 73a2e3326fe25d52d7067d946be509aeb0eea5c2) * Fix authorization check in domain blocks controller (cherry picked from commit 655bac2c3aed67eed868d32d1a5b02df4c46e6b4) * Fix error strings for domain blocks and email-domain blocks Corrected issue with non-error message used for Mastodon:NotPermittedError in Domain Blocks Corrected issue Domain Blocks using the Email Domain Blocks message on ActionContoller::ParameterMissing Corrected issue with Email Domain Blocks using the not_permitted string from "custom emojii's" * Ran i18n-tasks normalize to address test failure * Removed unused admin.export_domain_blocks.not_permitted string Removing unused string as indicated by Check i18n * Fix tests (cherry picked from commit 1ba41b34c1dfc872f0f19c146a38b0b21e4ee3f7) * Fix domain block export not exporting blocks with only media rejection (cherry picked from commit 1686374b2f2a380de63b616be56be7db918b35f0) * Fix various issues with domain block import - stop using Paperclip for processing domain allow/block imports - stop leaving temporary files - better error handling - assume CSV files are UTF-8-encoded (cherry picked from commit cad824d8f501b95377e4f0a957e5a00d517a1902) Co-authored-by: Levi Bard <taktaktaktaktaktaktaktaktaktak@gmail.com> Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-11-17 10:05:09 +00:00
def validate_data
return if data.nil?
errors.add(:data, I18n.t('imports.errors.over_rows_processing_limit', count: ROWS_PROCESSING_LIMIT)) if csv_row_count > ROWS_PROCESSING_LIMIT
Support for import/export of instance-level domain blocks/allows for 4.x w/ additional fixes (#20597) * Allow import/export of instance-level domain blocks/allows (#1754) * Allow import/export of instance-level domain blocks/allows. Fixes #15095 * Pacify circleci * Address simple code review feedback * Add headers to exported CSV * Extract common import/export functionality to AdminExportControllerConcern * Add additional fields to instance-blocked domain export * Address review feedback * Split instance domain block/allow import/export into separate pages/controllers * Address code review feedback * Pacify DeepSource * Work around Paperclip::HasAttachmentFile for Rails 6 * Fix deprecated API warning in export tests * Remove after_commit workaround (cherry picked from commit b8319d057824bc4bcdbac2cad16f97e264844168) * Add confirmation page when importing blocked domains (#1773) * Move glitch-soc-specific strings to glitch-soc-specific locale files * Add confirmation page when importing blocked domains (cherry picked from commit 73a2e3326fe25d52d7067d946be509aeb0eea5c2) * Fix authorization check in domain blocks controller (cherry picked from commit 655bac2c3aed67eed868d32d1a5b02df4c46e6b4) * Fix error strings for domain blocks and email-domain blocks Corrected issue with non-error message used for Mastodon:NotPermittedError in Domain Blocks Corrected issue Domain Blocks using the Email Domain Blocks message on ActionContoller::ParameterMissing Corrected issue with Email Domain Blocks using the not_permitted string from "custom emojii's" * Ran i18n-tasks normalize to address test failure * Removed unused admin.export_domain_blocks.not_permitted string Removing unused string as indicated by Check i18n * Fix tests (cherry picked from commit 1ba41b34c1dfc872f0f19c146a38b0b21e4ee3f7) * Fix domain block export not exporting blocks with only media rejection (cherry picked from commit 1686374b2f2a380de63b616be56be7db918b35f0) * Fix various issues with domain block import - stop using Paperclip for processing domain allow/block imports - stop leaving temporary files - better error handling - assume CSV files are UTF-8-encoded (cherry picked from commit cad824d8f501b95377e4f0a957e5a00d517a1902) Co-authored-by: Levi Bard <taktaktaktaktaktaktaktaktaktak@gmail.com> Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-11-17 10:05:09 +00:00
rescue CSV::MalformedCSVError => e
errors.add(:data, I18n.t('imports.errors.invalid_csv_file', error: e.message))
end
end