From ced99149d6e88f1327c6c9fd59697ac27c12db67 Mon Sep 17 00:00:00 2001 From: Rin Date: Mon, 21 Nov 2022 17:43:06 +1100 Subject: [PATCH 01/18] Add SETUP instructions --- SETUP.md | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 SETUP.md diff --git a/SETUP.md b/SETUP.md new file mode 100644 index 0000000000..a18cf5ca85 --- /dev/null +++ b/SETUP.md @@ -0,0 +1,78 @@ +# Setting up a dev environment + +## Assumptions + +You have a package manager, and your system is systemd-based. + +## Prerequisites + +## Nodejs + +Install nodejs via your package manager. We used v19.1.0 for this setup. + +## Ruby + +Install ruby via your package manager of choice. + +## Ruby Extras + +Your distro may package `bundle` and `irb` in separate packages (Arch does). If they are, install them too. +On Arch these are `ruby-bundler` and `ruby-irb`. + +## PostGreSQL + +Install postgres via your package manager. + +Enable and start the default `postgresql.service` + +## Redis + +Install redis via your package manager. + +Enable and start the default `redis.service` + +## Pre-Work + +In the following instructions, replace USER with your *nix user name. + +1. Give the postgres user rwx on your home folder. We used `sudo setfacl -m u:postgres:rwx /home/USER`. +2. Set up your user for postgres db creation with `sudo -u postgres createuser USER --createdb`. +3. Navigate to the root of this repo +4. Run `bundle config set --local path 'vendor/bundle'. This will store the all the ruby gems locally so that we can +avoid interfering with system config. +5. Run `bundle install`. +6. Run `yarn install`. +7. Run `bundle exec rake db:setup`. If this fails, you can use `bundle exec rake db:reset` to forcibly regenerate it. + +## Getting it running + +To make our lives easier, we'll use `foreman` to run the site, so use `gem install foreman` to get that going. + +Then: +1. Run `export RAILS\_ENV=development` and `export NODE\_ENV=development`. + a. Put these in your shell's .rc, or a script you can source if you want to skip this step in the future. +2. Run `bundle exec rake assets:precompile`. + a. If this explodes, complaining about `Hash`, you'll need to `export NODE\_OPTIONS=--openssl-legacy-provider`. Same + deal as the above. + b. After doing this, you will need to `bundle exec rake assets:clobber` and then re-run + `bundle exec rake assets:precompile`. +3. Run `foreman start` + + +# Updates/Troubleshooting + +## RubyVM/DebugInspector Issues + +Still unable to fix. Circumvent by removing `better\_errors` and `binding\_of\_caller` from Gemfile. +Happy to troubleshoot with someone better with Ruby than us >_<'/. + +## Webpack Issues +If Webpack compalins about being unable to find some assets or locales: + +1. yarn add webpack +2. git restore package.json yarn.lock +3. yarn install + +Then re-run `foreman`. No. We have no idea why this worked. + +If the above instructions don't work, please contact @Rin here, or @tammy@social.treehouse.systems. -- 2.41.0 From 4172c91de0b909e8ebb4c2e09228878b1f0e9bdf Mon Sep 17 00:00:00 2001 From: Rin Date: Mon, 21 Nov 2022 18:11:49 +1100 Subject: [PATCH 02/18] fix typo --- SETUP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SETUP.md b/SETUP.md index a18cf5ca85..91cc659c6c 100644 --- a/SETUP.md +++ b/SETUP.md @@ -38,7 +38,7 @@ In the following instructions, replace USER with your *nix user name. 1. Give the postgres user rwx on your home folder. We used `sudo setfacl -m u:postgres:rwx /home/USER`. 2. Set up your user for postgres db creation with `sudo -u postgres createuser USER --createdb`. 3. Navigate to the root of this repo -4. Run `bundle config set --local path 'vendor/bundle'. This will store the all the ruby gems locally so that we can +4. Run `bundle config set --local path 'vendor/bundle`. This will store the all the ruby gems locally so that we can avoid interfering with system config. 5. Run `bundle install`. 6. Run `yarn install`. -- 2.41.0 From 59b831de9be20dd40107702c431fac33973335e3 Mon Sep 17 00:00:00 2001 From: Rin Date: Mon, 21 Nov 2022 19:31:46 +1100 Subject: [PATCH 03/18] use local pg cluster instead --- .env.dev | 303 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ SETUP.md | 20 ++-- 2 files changed, 315 insertions(+), 8 deletions(-) create mode 100644 .env.dev diff --git a/.env.dev b/.env.dev new file mode 100644 index 0000000000..69d42711d6 --- /dev/null +++ b/.env.dev @@ -0,0 +1,303 @@ +# This is a sample configuration file. You can generate your configuration +# with the `rake mastodon:setup` interactive setup wizard, but to customize +# your setup even further, you'll need to edit it manually. This sample does +# not demonstrate all available configuration options. Please look at +# https://docs.joinmastodon.org/admin/config/ for the full documentation. + +# Note that this file accepts slightly different syntax depending on whether +# you are using `docker-compose` or not. In particular, if you use +# `docker-compose`, the value of each declared variable will be taken verbatim, +# including surrounding quotes. +# See: https://github.com/mastodon/mastodon/issues/16895 + +# Federation +# ---------- +# This identifies your server and cannot be changed safely later +# ---------- +LOCAL_DOMAIN=localhost + +# Use this only if you need to run mastodon on a different domain than the one used for federation. +# You can read more about this option on https://docs.joinmastodon.org/admin/config/#web-domain +# DO *NOT* USE THIS UNLESS YOU KNOW *EXACTLY* WHAT YOU ARE DOING. +# WEB_DOMAIN=mastodon.example.com + +# Use this if you want to have several aliases handler@example1.com +# handler@example2.com etc. for the same user. LOCAL_DOMAIN should not +# be added. Comma separated values +# ALTERNATE_DOMAINS=example1.com,example2.com + +# Use HTTP proxy for outgoing request (optional) +# http_proxy=http://gateway.local:8118 +# Access control for hidden service. +# ALLOW_ACCESS_TO_HIDDEN_SERVICE=true + +# Authorized fetch mode (optional) +# Require remote servers to authentify when fetching toots, see +# https://docs.joinmastodon.org/admin/config/#authorized_fetch +# AUTHORIZED_FETCH=true + +# Limited federation mode (optional) +# Only allow federation with specific domains, see +# https://docs.joinmastodon.org/admin/config/#whitelist_mode +# LIMITED_FEDERATION_MODE=true + +# Redis +# ----- +REDIS_HOST=localhost +REDIS_PORT=6379 + + +# PostgreSQL +# ---------- +DB_HOST=/var/run/postgresql +DB_USER=mastodon +DB_NAME=mastodon_development +DB_PASS= +DB_PORT=5432 + + +# Elasticsearch (optional) +# ------------------------ +#ES_ENABLED=true +#ES_HOST=localhost +#ES_PORT=9200 +# Authentication for ES (optional) +#ES_USER=elastic +#ES_PASS=password + + +# Secrets +# ------- +# Generate each with the `RAILS_ENV=production bundle exec rake secret` task (`docker-compose run --rm web bundle exec rake secret` if you use docker compose) +# ------- +#SECRET_KEY_BASE= +#OTP_SECRET= + + +# Web Push +# -------- +# Generate with `rake mastodon:webpush:generate_vapid_key` (first is the private key, second is the public one) +# You should only generate this once per instance. If you later decide to change it, all push subscription will +# be invalidated, requiring the users to access the website again to resubscribe. +# -------- +#VAPID_PRIVATE_KEY= +#VAPID_PUBLIC_KEY= + + +# Registrations +# ------------- + +# Single user mode will disable registrations and redirect frontpage to the first profile +# SINGLE_USER_MODE=true + +# Prevent registrations with following e-mail domains +# EMAIL_DOMAIN_DENYLIST=example1.com|example2.de|etc + +# Only allow registrations with the following e-mail domains +# EMAIL_DOMAIN_ALLOWLIST=example1.com|example2.de|etc + +#TODO move this +# Optionally change default language +# DEFAULT_LOCALE=de + + +# Sending mail +# ------------ +#SMTP_SERVER=smtp.mailgun.org +#SMTP_PORT=587 +#SMTP_LOGIN= +#SMTP_PASSWORD= +#SMTP_FROM_ADDRESS=notifications@example.com + + +# File storage (optional) +# ----------------------- +# The attachment host must allow cross origin request from WEB_DOMAIN or +# LOCAL_DOMAIN if WEB_DOMAIN is not set. For example, the server may have the +# following header field: +# Access-Control-Allow-Origin: https://192.168.1.123:9000/ +# ----------------------- +#S3_ENABLED=true +#S3_BUCKET=files.example.com +#AWS_ACCESS_KEY_ID= +#AWS_SECRET_ACCESS_KEY= +#S3_ALIAS_HOST=files.example.com + +# Swift (optional) +# The attachment host must allow cross origin request - see the description +# above. +# SWIFT_ENABLED=true +# SWIFT_USERNAME= +# For Keystone V3, the value for SWIFT_TENANT should be the project name +# SWIFT_TENANT= +# SWIFT_PASSWORD= +# Some OpenStack V3 providers require PROJECT_ID (optional) +# SWIFT_PROJECT_ID= +# Keystone V2 and V3 URLs are supported. Use a V3 URL if possible to avoid +# issues with token rate-limiting during high load. +# SWIFT_AUTH_URL= +# SWIFT_CONTAINER= +# SWIFT_OBJECT_URL= +# SWIFT_REGION= +# Defaults to 'default' +# SWIFT_DOMAIN_NAME= +# Defaults to 60 seconds. Set to 0 to disable +# SWIFT_CACHE_TTL= + +# Optional asset host for multi-server setups +# The asset host must allow cross origin request from WEB_DOMAIN or LOCAL_DOMAIN +# if WEB_DOMAIN is not set. For example, the server may have the +# following header field: +# Access-Control-Allow-Origin: https://example.com/ +# CDN_HOST=https://assets.example.com + +# Optional list of hosts that are allowed to serve media for your instance +# This is useful if you include external media in your custom CSS or about page, +# or if your data storage provider makes use of redirects to other domains. +# EXTRA_DATA_HOSTS=https://data.example1.com|https://data.example2.com + +# Optional alias for S3 (e.g. to serve files on a custom domain, possibly using Cloudfront or Cloudflare) +# S3_ALIAS_HOST= + +# Streaming API integration +# STREAMING_API_BASE_URL= + + +# External authentication (optional) +# ---------------------------------- +# LDAP authentication (optional) +# LDAP_ENABLED=true +# LDAP_HOST=localhost +# LDAP_PORT=389 +# LDAP_METHOD=simple_tls +# LDAP_BASE= +# LDAP_BIND_DN= +# LDAP_PASSWORD= +# LDAP_UID=cn +# LDAP_MAIL=mail +# LDAP_SEARCH_FILTER=(|(%{uid}=%{email})(%{mail}=%{email})) +# LDAP_UID_CONVERSION_ENABLED=true +# LDAP_UID_CONVERSION_SEARCH=., - +# LDAP_UID_CONVERSION_REPLACE=_ + +# PAM authentication (optional) +# PAM authentication uses for the email generation the "email" pam variable +# and optional as fallback PAM_DEFAULT_SUFFIX +# The pam environment variable "email" is provided by: +# https://github.com/devkral/pam_email_extractor +# PAM_ENABLED=true +# Fallback email domain for email address generation (LOCAL_DOMAIN by default) +# PAM_EMAIL_DOMAIN=example.com +# Name of the pam service (pam "auth" section is evaluated) +# PAM_DEFAULT_SERVICE=rpam +# Name of the pam service used for checking if an user can register (pam "account" section is evaluated) (nil (disabled) by default) +# PAM_CONTROLLED_SERVICE=rpam + +# Global OAuth settings (optional) : +# If you have only one strategy, you may want to enable this +# OAUTH_REDIRECT_AT_SIGN_IN=true + +# Optional CAS authentication (cf. omniauth-cas) : +# CAS_ENABLED=true +# CAS_URL=https://sso.myserver.com/ +# CAS_HOST=sso.myserver.com/ +# CAS_PORT=443 +# CAS_SSL=true +# CAS_VALIDATE_URL= +# CAS_CALLBACK_URL= +# CAS_LOGOUT_URL= +# CAS_LOGIN_URL= +# CAS_UID_FIELD='user' +# CAS_CA_PATH= +# CAS_DISABLE_SSL_VERIFICATION=false +# CAS_UID_KEY='user' +# CAS_NAME_KEY='name' +# CAS_EMAIL_KEY='email' +# CAS_NICKNAME_KEY='nickname' +# CAS_FIRST_NAME_KEY='firstname' +# CAS_LAST_NAME_KEY='lastname' +# CAS_LOCATION_KEY='location' +# CAS_IMAGE_KEY='image' +# CAS_PHONE_KEY='phone' + +# Optional SAML authentication (cf. omniauth-saml) +# SAML_ENABLED=true +# SAML_ACS_URL=http://localhost:3000/auth/auth/saml/callback +# SAML_ISSUER=https://example.com +# SAML_IDP_SSO_TARGET_URL=https://idp.testshib.org/idp/profile/SAML2/Redirect/SSO +# SAML_IDP_CERT= +# SAML_IDP_CERT_FINGERPRINT= +# SAML_NAME_IDENTIFIER_FORMAT= +# SAML_CERT= +# SAML_PRIVATE_KEY= +# SAML_SECURITY_WANT_ASSERTION_SIGNED=true +# SAML_SECURITY_WANT_ASSERTION_ENCRYPTED=true +# SAML_SECURITY_ASSUME_EMAIL_IS_VERIFIED=true +# SAML_ATTRIBUTES_STATEMENTS_UID="urn:oid:0.9.2342.19200300.100.1.1" +# SAML_ATTRIBUTES_STATEMENTS_EMAIL="urn:oid:1.3.6.1.4.1.5923.1.1.1.6" +# SAML_ATTRIBUTES_STATEMENTS_FULL_NAME="urn:oid:2.16.840.1.113730.3.1.241" +# SAML_ATTRIBUTES_STATEMENTS_FIRST_NAME="urn:oid:2.5.4.42" +# SAML_ATTRIBUTES_STATEMENTS_LAST_NAME="urn:oid:2.5.4.4" +# SAML_UID_ATTRIBUTE="urn:oid:0.9.2342.19200300.100.1.1" +# SAML_ATTRIBUTES_STATEMENTS_VERIFIED= +# SAML_ATTRIBUTES_STATEMENTS_VERIFIED_EMAIL= + + +# Custom settings +# --------------- +# Various ways to customize Mastodon's behavior +# --------------- + +# Maximum allowed character count +MAX_TOOT_CHARS=500 + +# Maximum number of pinned posts +MAX_PINNED_TOOTS=5 + +# Maximum allowed bio characters +MAX_BIO_CHARS=500 + +# Maximim number of profile fields allowed +MAX_PROFILE_FIELDS=4 + +# Maximum allowed display name characters +MAX_DISPLAY_NAME_CHARS=30 + +# Maximum allowed poll options +MAX_POLL_OPTIONS=5 + +# Maximum allowed poll option characters +MAX_POLL_OPTION_CHARS=100 + +# Maximum image and video/audio upload sizes +# Units are in bytes +# 1048576 bytes equals 1 megabyte +# MAX_IMAGE_SIZE=8388608 +# MAX_VIDEO_SIZE=41943040 + +# Maximum search results to display +# Only relevant when elasticsearch is installed +# MAX_SEARCH_RESULTS=20 + +# Maximum hashtags to display +# Customize the number of hashtags shown in 'Explore' +# MAX_TRENDING_TAGS=10 + +# Maximum custom emoji file sizes +# If undefined or smaller than MAX_EMOJI_SIZE, the value +# of MAX_EMOJI_SIZE will be used for MAX_REMOTE_EMOJI_SIZE +# Units are in bytes +# MAX_EMOJI_SIZE=262144 +# MAX_REMOTE_EMOJI_SIZE=262144 + +# Optional hCaptcha support +# HCAPTCHA_SECRET_KEY= +# HCAPTCHA_SITE_KEY= + +# IP and session retention +# ----------------------- +# Make sure to modify the scheduling of ip_cleanup_scheduler in config/sidekiq.yml +# to be less than daily if you lower IP_RETENTION_PERIOD below two days (172800). +# ----------------------- +IP_RETENTION_PERIOD=31556952 +SESSION_RETENTION_PERIOD=31556952 diff --git a/SETUP.md b/SETUP.md index 91cc659c6c..96bab94b51 100644 --- a/SETUP.md +++ b/SETUP.md @@ -23,8 +23,6 @@ On Arch these are `ruby-bundler` and `ruby-irb`. Install postgres via your package manager. -Enable and start the default `postgresql.service` - ## Redis Install redis via your package manager. @@ -35,13 +33,19 @@ Enable and start the default `redis.service` In the following instructions, replace USER with your *nix user name. -1. Give the postgres user rwx on your home folder. We used `sudo setfacl -m u:postgres:rwx /home/USER`. -2. Set up your user for postgres db creation with `sudo -u postgres createuser USER --createdb`. -3. Navigate to the root of this repo +1. Add yourself to the postgres group with `sudo usermod -a -G postgres USER`. You'll need to log out and back in to +update your groups. +2. Run `sudo mkdir /run/postgresql` to create said folder if it doesn't exist. +3. Run `sudo chown postgres:postgres /run/postgresql` to change the owner to postgres. +4. Run `sudo chmod g+w /run/postgresql` to allow `postgres` group members to write to the folder. +1. Navigate to the root of this repo. +2. Set up a local DB cluster with `pg_ctl -D data/postgres15 initdb -o '-U mastodon --auth-host=trust'`. +3. Run it with `pg_ctl -D data/postgres15 start`. 4. Run `bundle config set --local path 'vendor/bundle`. This will store the all the ruby gems locally so that we can avoid interfering with system config. 5. Run `bundle install`. 6. Run `yarn install`. +1. Run `export $(grep -v '^#' .env.dev | xargs)` to source in our dev vars. You may want to alias this. 7. Run `bundle exec rake db:setup`. If this fails, you can use `bundle exec rake db:reset` to forcibly regenerate it. ## Getting it running @@ -49,10 +53,10 @@ avoid interfering with system config. To make our lives easier, we'll use `foreman` to run the site, so use `gem install foreman` to get that going. Then: -1. Run `export RAILS\_ENV=development` and `export NODE\_ENV=development`. +1. Run `export RAILS_ENV=development` and `export NODE_ENV=development`. a. Put these in your shell's .rc, or a script you can source if you want to skip this step in the future. 2. Run `bundle exec rake assets:precompile`. - a. If this explodes, complaining about `Hash`, you'll need to `export NODE\_OPTIONS=--openssl-legacy-provider`. Same + a. If this explodes, complaining about `Hash`, you'll need to `export NODE_OPTIONS=--openssl-legacy-provider`. Same deal as the above. b. After doing this, you will need to `bundle exec rake assets:clobber` and then re-run `bundle exec rake assets:precompile`. @@ -63,7 +67,7 @@ Then: ## RubyVM/DebugInspector Issues -Still unable to fix. Circumvent by removing `better\_errors` and `binding\_of\_caller` from Gemfile. +Still unable to fix. Circumvent by removing `better_errors` and `binding_of_caller` from Gemfile. Happy to troubleshoot with someone better with Ruby than us >_<'/. ## Webpack Issues -- 2.41.0 From 74c54af9046474ad1550a17770d64508b0de8366 Mon Sep 17 00:00:00 2001 From: Rin Date: Mon, 21 Nov 2022 19:33:13 +1100 Subject: [PATCH 04/18] update gitignore to ignore the pg db we created --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index 7d76b82751..75b04cb99b 100644 --- a/.gitignore +++ b/.gitignore @@ -12,6 +12,9 @@ /db/*.sqlite3 /db/*.sqlite3-journal +# Ignore the dev postgres db we created. +/data/postgres15/* + # Ignore all logfiles and tempfiles. .eslintcache /log/* -- 2.41.0 From 2790045c7e27eb47a17e29d26096fae02ada118e Mon Sep 17 00:00:00 2001 From: Rin Date: Mon, 21 Nov 2022 21:06:35 +1100 Subject: [PATCH 05/18] Reduce .env file to minimal config --- .env.dev | 299 ------------------------------------------------------- SETUP.md | 4 +- 2 files changed, 2 insertions(+), 301 deletions(-) diff --git a/.env.dev b/.env.dev index 69d42711d6..9ea71b721f 100644 --- a/.env.dev +++ b/.env.dev @@ -1,303 +1,4 @@ -# This is a sample configuration file. You can generate your configuration -# with the `rake mastodon:setup` interactive setup wizard, but to customize -# your setup even further, you'll need to edit it manually. This sample does -# not demonstrate all available configuration options. Please look at -# https://docs.joinmastodon.org/admin/config/ for the full documentation. - -# Note that this file accepts slightly different syntax depending on whether -# you are using `docker-compose` or not. In particular, if you use -# `docker-compose`, the value of each declared variable will be taken verbatim, -# including surrounding quotes. -# See: https://github.com/mastodon/mastodon/issues/16895 - -# Federation -# ---------- -# This identifies your server and cannot be changed safely later -# ---------- LOCAL_DOMAIN=localhost - -# Use this only if you need to run mastodon on a different domain than the one used for federation. -# You can read more about this option on https://docs.joinmastodon.org/admin/config/#web-domain -# DO *NOT* USE THIS UNLESS YOU KNOW *EXACTLY* WHAT YOU ARE DOING. -# WEB_DOMAIN=mastodon.example.com - -# Use this if you want to have several aliases handler@example1.com -# handler@example2.com etc. for the same user. LOCAL_DOMAIN should not -# be added. Comma separated values -# ALTERNATE_DOMAINS=example1.com,example2.com - -# Use HTTP proxy for outgoing request (optional) -# http_proxy=http://gateway.local:8118 -# Access control for hidden service. -# ALLOW_ACCESS_TO_HIDDEN_SERVICE=true - -# Authorized fetch mode (optional) -# Require remote servers to authentify when fetching toots, see -# https://docs.joinmastodon.org/admin/config/#authorized_fetch -# AUTHORIZED_FETCH=true - -# Limited federation mode (optional) -# Only allow federation with specific domains, see -# https://docs.joinmastodon.org/admin/config/#whitelist_mode -# LIMITED_FEDERATION_MODE=true - -# Redis -# ----- -REDIS_HOST=localhost -REDIS_PORT=6379 - - -# PostgreSQL -# ---------- DB_HOST=/var/run/postgresql DB_USER=mastodon DB_NAME=mastodon_development -DB_PASS= -DB_PORT=5432 - - -# Elasticsearch (optional) -# ------------------------ -#ES_ENABLED=true -#ES_HOST=localhost -#ES_PORT=9200 -# Authentication for ES (optional) -#ES_USER=elastic -#ES_PASS=password - - -# Secrets -# ------- -# Generate each with the `RAILS_ENV=production bundle exec rake secret` task (`docker-compose run --rm web bundle exec rake secret` if you use docker compose) -# ------- -#SECRET_KEY_BASE= -#OTP_SECRET= - - -# Web Push -# -------- -# Generate with `rake mastodon:webpush:generate_vapid_key` (first is the private key, second is the public one) -# You should only generate this once per instance. If you later decide to change it, all push subscription will -# be invalidated, requiring the users to access the website again to resubscribe. -# -------- -#VAPID_PRIVATE_KEY= -#VAPID_PUBLIC_KEY= - - -# Registrations -# ------------- - -# Single user mode will disable registrations and redirect frontpage to the first profile -# SINGLE_USER_MODE=true - -# Prevent registrations with following e-mail domains -# EMAIL_DOMAIN_DENYLIST=example1.com|example2.de|etc - -# Only allow registrations with the following e-mail domains -# EMAIL_DOMAIN_ALLOWLIST=example1.com|example2.de|etc - -#TODO move this -# Optionally change default language -# DEFAULT_LOCALE=de - - -# Sending mail -# ------------ -#SMTP_SERVER=smtp.mailgun.org -#SMTP_PORT=587 -#SMTP_LOGIN= -#SMTP_PASSWORD= -#SMTP_FROM_ADDRESS=notifications@example.com - - -# File storage (optional) -# ----------------------- -# The attachment host must allow cross origin request from WEB_DOMAIN or -# LOCAL_DOMAIN if WEB_DOMAIN is not set. For example, the server may have the -# following header field: -# Access-Control-Allow-Origin: https://192.168.1.123:9000/ -# ----------------------- -#S3_ENABLED=true -#S3_BUCKET=files.example.com -#AWS_ACCESS_KEY_ID= -#AWS_SECRET_ACCESS_KEY= -#S3_ALIAS_HOST=files.example.com - -# Swift (optional) -# The attachment host must allow cross origin request - see the description -# above. -# SWIFT_ENABLED=true -# SWIFT_USERNAME= -# For Keystone V3, the value for SWIFT_TENANT should be the project name -# SWIFT_TENANT= -# SWIFT_PASSWORD= -# Some OpenStack V3 providers require PROJECT_ID (optional) -# SWIFT_PROJECT_ID= -# Keystone V2 and V3 URLs are supported. Use a V3 URL if possible to avoid -# issues with token rate-limiting during high load. -# SWIFT_AUTH_URL= -# SWIFT_CONTAINER= -# SWIFT_OBJECT_URL= -# SWIFT_REGION= -# Defaults to 'default' -# SWIFT_DOMAIN_NAME= -# Defaults to 60 seconds. Set to 0 to disable -# SWIFT_CACHE_TTL= - -# Optional asset host for multi-server setups -# The asset host must allow cross origin request from WEB_DOMAIN or LOCAL_DOMAIN -# if WEB_DOMAIN is not set. For example, the server may have the -# following header field: -# Access-Control-Allow-Origin: https://example.com/ -# CDN_HOST=https://assets.example.com - -# Optional list of hosts that are allowed to serve media for your instance -# This is useful if you include external media in your custom CSS or about page, -# or if your data storage provider makes use of redirects to other domains. -# EXTRA_DATA_HOSTS=https://data.example1.com|https://data.example2.com - -# Optional alias for S3 (e.g. to serve files on a custom domain, possibly using Cloudfront or Cloudflare) -# S3_ALIAS_HOST= - -# Streaming API integration -# STREAMING_API_BASE_URL= - - -# External authentication (optional) -# ---------------------------------- -# LDAP authentication (optional) -# LDAP_ENABLED=true -# LDAP_HOST=localhost -# LDAP_PORT=389 -# LDAP_METHOD=simple_tls -# LDAP_BASE= -# LDAP_BIND_DN= -# LDAP_PASSWORD= -# LDAP_UID=cn -# LDAP_MAIL=mail -# LDAP_SEARCH_FILTER=(|(%{uid}=%{email})(%{mail}=%{email})) -# LDAP_UID_CONVERSION_ENABLED=true -# LDAP_UID_CONVERSION_SEARCH=., - -# LDAP_UID_CONVERSION_REPLACE=_ - -# PAM authentication (optional) -# PAM authentication uses for the email generation the "email" pam variable -# and optional as fallback PAM_DEFAULT_SUFFIX -# The pam environment variable "email" is provided by: -# https://github.com/devkral/pam_email_extractor -# PAM_ENABLED=true -# Fallback email domain for email address generation (LOCAL_DOMAIN by default) -# PAM_EMAIL_DOMAIN=example.com -# Name of the pam service (pam "auth" section is evaluated) -# PAM_DEFAULT_SERVICE=rpam -# Name of the pam service used for checking if an user can register (pam "account" section is evaluated) (nil (disabled) by default) -# PAM_CONTROLLED_SERVICE=rpam - -# Global OAuth settings (optional) : -# If you have only one strategy, you may want to enable this -# OAUTH_REDIRECT_AT_SIGN_IN=true - -# Optional CAS authentication (cf. omniauth-cas) : -# CAS_ENABLED=true -# CAS_URL=https://sso.myserver.com/ -# CAS_HOST=sso.myserver.com/ -# CAS_PORT=443 -# CAS_SSL=true -# CAS_VALIDATE_URL= -# CAS_CALLBACK_URL= -# CAS_LOGOUT_URL= -# CAS_LOGIN_URL= -# CAS_UID_FIELD='user' -# CAS_CA_PATH= -# CAS_DISABLE_SSL_VERIFICATION=false -# CAS_UID_KEY='user' -# CAS_NAME_KEY='name' -# CAS_EMAIL_KEY='email' -# CAS_NICKNAME_KEY='nickname' -# CAS_FIRST_NAME_KEY='firstname' -# CAS_LAST_NAME_KEY='lastname' -# CAS_LOCATION_KEY='location' -# CAS_IMAGE_KEY='image' -# CAS_PHONE_KEY='phone' - -# Optional SAML authentication (cf. omniauth-saml) -# SAML_ENABLED=true -# SAML_ACS_URL=http://localhost:3000/auth/auth/saml/callback -# SAML_ISSUER=https://example.com -# SAML_IDP_SSO_TARGET_URL=https://idp.testshib.org/idp/profile/SAML2/Redirect/SSO -# SAML_IDP_CERT= -# SAML_IDP_CERT_FINGERPRINT= -# SAML_NAME_IDENTIFIER_FORMAT= -# SAML_CERT= -# SAML_PRIVATE_KEY= -# SAML_SECURITY_WANT_ASSERTION_SIGNED=true -# SAML_SECURITY_WANT_ASSERTION_ENCRYPTED=true -# SAML_SECURITY_ASSUME_EMAIL_IS_VERIFIED=true -# SAML_ATTRIBUTES_STATEMENTS_UID="urn:oid:0.9.2342.19200300.100.1.1" -# SAML_ATTRIBUTES_STATEMENTS_EMAIL="urn:oid:1.3.6.1.4.1.5923.1.1.1.6" -# SAML_ATTRIBUTES_STATEMENTS_FULL_NAME="urn:oid:2.16.840.1.113730.3.1.241" -# SAML_ATTRIBUTES_STATEMENTS_FIRST_NAME="urn:oid:2.5.4.42" -# SAML_ATTRIBUTES_STATEMENTS_LAST_NAME="urn:oid:2.5.4.4" -# SAML_UID_ATTRIBUTE="urn:oid:0.9.2342.19200300.100.1.1" -# SAML_ATTRIBUTES_STATEMENTS_VERIFIED= -# SAML_ATTRIBUTES_STATEMENTS_VERIFIED_EMAIL= - - -# Custom settings -# --------------- -# Various ways to customize Mastodon's behavior -# --------------- - -# Maximum allowed character count -MAX_TOOT_CHARS=500 - -# Maximum number of pinned posts -MAX_PINNED_TOOTS=5 - -# Maximum allowed bio characters -MAX_BIO_CHARS=500 - -# Maximim number of profile fields allowed -MAX_PROFILE_FIELDS=4 - -# Maximum allowed display name characters -MAX_DISPLAY_NAME_CHARS=30 - -# Maximum allowed poll options -MAX_POLL_OPTIONS=5 - -# Maximum allowed poll option characters -MAX_POLL_OPTION_CHARS=100 - -# Maximum image and video/audio upload sizes -# Units are in bytes -# 1048576 bytes equals 1 megabyte -# MAX_IMAGE_SIZE=8388608 -# MAX_VIDEO_SIZE=41943040 - -# Maximum search results to display -# Only relevant when elasticsearch is installed -# MAX_SEARCH_RESULTS=20 - -# Maximum hashtags to display -# Customize the number of hashtags shown in 'Explore' -# MAX_TRENDING_TAGS=10 - -# Maximum custom emoji file sizes -# If undefined or smaller than MAX_EMOJI_SIZE, the value -# of MAX_EMOJI_SIZE will be used for MAX_REMOTE_EMOJI_SIZE -# Units are in bytes -# MAX_EMOJI_SIZE=262144 -# MAX_REMOTE_EMOJI_SIZE=262144 - -# Optional hCaptcha support -# HCAPTCHA_SECRET_KEY= -# HCAPTCHA_SITE_KEY= - -# IP and session retention -# ----------------------- -# Make sure to modify the scheduling of ip_cleanup_scheduler in config/sidekiq.yml -# to be less than daily if you lower IP_RETENTION_PERIOD below two days (172800). -# ----------------------- -IP_RETENTION_PERIOD=31556952 -SESSION_RETENTION_PERIOD=31556952 diff --git a/SETUP.md b/SETUP.md index 96bab94b51..431a174401 100644 --- a/SETUP.md +++ b/SETUP.md @@ -29,7 +29,7 @@ Install redis via your package manager. Enable and start the default `redis.service` -## Pre-Work +## Setting up the Environment In the following instructions, replace USER with your *nix user name. @@ -48,7 +48,7 @@ avoid interfering with system config. 1. Run `export $(grep -v '^#' .env.dev | xargs)` to source in our dev vars. You may want to alias this. 7. Run `bundle exec rake db:setup`. If this fails, you can use `bundle exec rake db:reset` to forcibly regenerate it. -## Getting it running +## Running Mastodon To make our lives easier, we'll use `foreman` to run the site, so use `gem install foreman` to get that going. -- 2.41.0 From 8a701c5ac31f8adadf2ee6490923bedce50ace84 Mon Sep 17 00:00:00 2001 From: Rin Date: Mon, 21 Nov 2022 21:27:58 +1100 Subject: [PATCH 06/18] Fix numbering <_< --- SETUP.md | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/SETUP.md b/SETUP.md index 431a174401..bea067f2a4 100644 --- a/SETUP.md +++ b/SETUP.md @@ -38,29 +38,27 @@ update your groups. 2. Run `sudo mkdir /run/postgresql` to create said folder if it doesn't exist. 3. Run `sudo chown postgres:postgres /run/postgresql` to change the owner to postgres. 4. Run `sudo chmod g+w /run/postgresql` to allow `postgres` group members to write to the folder. -1. Navigate to the root of this repo. -2. Set up a local DB cluster with `pg_ctl -D data/postgres15 initdb -o '-U mastodon --auth-host=trust'`. -3. Run it with `pg_ctl -D data/postgres15 start`. -4. Run `bundle config set --local path 'vendor/bundle`. This will store the all the ruby gems locally so that we can -avoid interfering with system config. -5. Run `bundle install`. -6. Run `yarn install`. -1. Run `export $(grep -v '^#' .env.dev | xargs)` to source in our dev vars. You may want to alias this. -7. Run `bundle exec rake db:setup`. If this fails, you can use `bundle exec rake db:reset` to forcibly regenerate it. +5. Navigate to the root of this repo. +6. Set up a local DB cluster with `pg_ctl -D data/postgres15 initdb -o '-U mastodon --auth-host=trust'`. +7. Run it with `pg_ctl -D data/postgres15 start`. +8. Run `bundle config set --local path 'vendor/bundle`. This will store the all the ruby gems locally so that we can + avoid interfering with system config. +9. Run `bundle install`. +10. Run `yarn install`. +11. Run `export $(grep -v '^#' .env.dev | xargs)` to source in our dev vars. You may want to alias this. +12. Run `bundle exec rake db:setup`. If this fails, you can use `bundle exec rake db:reset` to forcibly regenerate it. ## Running Mastodon -To make our lives easier, we'll use `foreman` to run the site, so use `gem install foreman` to get that going. - -Then: -1. Run `export RAILS_ENV=development` and `export NODE_ENV=development`. +1. To make our lives easier, we'll use `foreman` to run the site, so use `gem install foreman`. +2. Run `export RAILS_ENV=development` and `export NODE_ENV=development`. a. Put these in your shell's .rc, or a script you can source if you want to skip this step in the future. -2. Run `bundle exec rake assets:precompile`. +3. Run `bundle exec rake assets:precompile`. a. If this explodes, complaining about `Hash`, you'll need to `export NODE_OPTIONS=--openssl-legacy-provider`. Same deal as the above. b. After doing this, you will need to `bundle exec rake assets:clobber` and then re-run `bundle exec rake assets:precompile`. -3. Run `foreman start` +4. Run `foreman start` # Updates/Troubleshooting -- 2.41.0 From ca98b0da8448c3644d1b1ce093c8dc0f032f86ab Mon Sep 17 00:00:00 2001 From: Rin Date: Wed, 23 Nov 2022 19:10:31 +1100 Subject: [PATCH 07/18] Add .env.ENV.local to .gitignore --- .gitignore | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 75b04cb99b..20f5fc0585 100644 --- a/.gitignore +++ b/.gitignore @@ -4,6 +4,11 @@ # or operating system, you probably want to add a global ignore instead: # git config --global core.excludesfile '~/.gitignore_global' +# Ignore local dotenv overrides +.env.development.local +.env.test.local +.env.production.local + # Ignore bundler config and downloaded libraries. /.bundle /vendor/bundle @@ -13,7 +18,7 @@ /db/*.sqlite3-journal # Ignore the dev postgres db we created. -/data/postgres15/* +/data/postgres*/* # Ignore all logfiles and tempfiles. .eslintcache -- 2.41.0 From 17e49cf09816659e473f8aa57725753bf822e3b6 Mon Sep 17 00:00:00 2001 From: Rin Date: Wed, 23 Nov 2022 19:22:32 +1100 Subject: [PATCH 08/18] rename env file to conform --- .env.dev => .env.development | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename .env.dev => .env.development (100%) diff --git a/.env.dev b/.env.development similarity index 100% rename from .env.dev rename to .env.development -- 2.41.0 From 842498649da2eb58abae24bec4db869883c0977d Mon Sep 17 00:00:00 2001 From: Rin Date: Wed, 23 Nov 2022 19:25:17 +1100 Subject: [PATCH 09/18] Moderately curse the env file to contain everything in the project --- .env.development | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.env.development b/.env.development index 9ea71b721f..e4987e4970 100644 --- a/.env.development +++ b/.env.development @@ -1,4 +1,4 @@ LOCAL_DOMAIN=localhost -DB_HOST=/var/run/postgresql +DB_HOST=$(pwd)/data/postgres15 DB_USER=mastodon DB_NAME=mastodon_development -- 2.41.0 From 401a51b67b61bfc11adc90665ad32b8bc1848f6b Mon Sep 17 00:00:00 2001 From: Rin Date: Wed, 23 Nov 2022 19:31:10 +1100 Subject: [PATCH 10/18] Update instructions - make everything self contained --- SETUP.md | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/SETUP.md b/SETUP.md index bea067f2a4..77591eacc0 100644 --- a/SETUP.md +++ b/SETUP.md @@ -35,18 +35,15 @@ In the following instructions, replace USER with your *nix user name. 1. Add yourself to the postgres group with `sudo usermod -a -G postgres USER`. You'll need to log out and back in to update your groups. -2. Run `sudo mkdir /run/postgresql` to create said folder if it doesn't exist. -3. Run `sudo chown postgres:postgres /run/postgresql` to change the owner to postgres. -4. Run `sudo chmod g+w /run/postgresql` to allow `postgres` group members to write to the folder. -5. Navigate to the root of this repo. -6. Set up a local DB cluster with `pg_ctl -D data/postgres15 initdb -o '-U mastodon --auth-host=trust'`. -7. Run it with `pg_ctl -D data/postgres15 start`. -8. Run `bundle config set --local path 'vendor/bundle`. This will store the all the ruby gems locally so that we can +2. Navigate to the root of this repo. +2. Set up a local DB cluster with `pg_ctl -D data/postgres15 initdb -o '-U mastodon --auth-host=trust'`. +3. Add the line `unix_socket_directories='.'` to `data/postgres15/postgresql.conf`. +4. Run the DB with `pg_ctl -D data/postgres15 start`. +5. Run `bundle config set --local path 'vendor/bundle`. This will store the all the ruby gems locally so that we can avoid interfering with system config. -9. Run `bundle install`. -10. Run `yarn install`. -11. Run `export $(grep -v '^#' .env.dev | xargs)` to source in our dev vars. You may want to alias this. -12. Run `bundle exec rake db:setup`. If this fails, you can use `bundle exec rake db:reset` to forcibly regenerate it. +6. Run `bundle install`. +7. Run `yarn install`. +8. Run `bundle exec rake db:setup`. If this fails, you can use `bundle exec rake db:reset` to forcibly regenerate it. ## Running Mastodon @@ -70,6 +67,12 @@ Happy to troubleshoot with someone better with Ruby than us >_<'/. ## Webpack Issues If Webpack compalins about being unable to find some assets or locales: +Try: + +1. `rm -rf node_modules` +2. `yarn install` + +If this doesn't help, try: 1. yarn add webpack 2. git restore package.json yarn.lock -- 2.41.0 From 8e4da8677c10b68decc35bd5d0f8433c3e5e8168 Mon Sep 17 00:00:00 2001 From: Rin Date: Wed, 23 Nov 2022 19:34:20 +1100 Subject: [PATCH 11/18] Fix headings --- SETUP.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/SETUP.md b/SETUP.md index 77591eacc0..22b59079e1 100644 --- a/SETUP.md +++ b/SETUP.md @@ -6,24 +6,24 @@ You have a package manager, and your system is systemd-based. ## Prerequisites -## Nodejs +### Nodejs Install nodejs via your package manager. We used v19.1.0 for this setup. -## Ruby +### Ruby Install ruby via your package manager of choice. -## Ruby Extras +### Ruby Extras Your distro may package `bundle` and `irb` in separate packages (Arch does). If they are, install them too. On Arch these are `ruby-bundler` and `ruby-irb`. -## PostGreSQL +### PostGreSQL Install postgres via your package manager. -## Redis +### Redis Install redis via your package manager. -- 2.41.0 From f4b48ca5ee44d543bb751fad8ab9d20876aebda6 Mon Sep 17 00:00:00 2001 From: Rin Date: Wed, 23 Nov 2022 19:35:13 +1100 Subject: [PATCH 12/18] Command style consistency --- SETUP.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/SETUP.md b/SETUP.md index 22b59079e1..5db498b96f 100644 --- a/SETUP.md +++ b/SETUP.md @@ -74,9 +74,9 @@ Try: If this doesn't help, try: -1. yarn add webpack -2. git restore package.json yarn.lock -3. yarn install +1. `yarn add webpack` +2. `git restore package.json yarn.lock` +3. `yarn install` Then re-run `foreman`. No. We have no idea why this worked. -- 2.41.0 From 17ba99b15741e8c22bcd0109f72d26919a3fc0c9 Mon Sep 17 00:00:00 2001 From: Kouhai Date: Thu, 24 Nov 2022 18:57:24 -0600 Subject: [PATCH 13/18] rewrite instructions for a self-contained dev env --- .env.development | 6 ++- .env.test | 5 ++ SETUP.md | 129 +++++++++++++++++++++++++++++++++-------------- redis-dev.conf | 12 +++++ 4 files changed, 113 insertions(+), 39 deletions(-) create mode 100644 redis-dev.conf diff --git a/.env.development b/.env.development index e4987e4970..1a11e918ef 100644 --- a/.env.development +++ b/.env.development @@ -1,4 +1,6 @@ LOCAL_DOMAIN=localhost -DB_HOST=$(pwd)/data/postgres15 + +DB_HOST=$(pwd)/data/postgres DB_USER=mastodon -DB_NAME=mastodon_development +DB_NAME=mastodon_dev +REDIS_URL=redis://$(pwd)/data/redis/redis-dev.sock diff --git a/.env.test b/.env.test index 761d0d9210..e3706868ca 100644 --- a/.env.test +++ b/.env.test @@ -3,3 +3,8 @@ NODE_ENV=tests # Federation LOCAL_DOMAIN=cb6e6126.ngrok.io LOCAL_HTTPS=true + +DB_HOST=$(pwd)/data/postgres +DB_USER=mastodon +DB_NAME=mastodon_dev +REDIS_URL=redis://$(pwd)/data/redis/redis-test.sock diff --git a/SETUP.md b/SETUP.md index 5db498b96f..1425e79e1e 100644 --- a/SETUP.md +++ b/SETUP.md @@ -1,62 +1,117 @@ # Setting up a dev environment -## Assumptions - -You have a package manager, and your system is systemd-based. - ## Prerequisites -### Nodejs +Mastodon development requires the following: -Install nodejs via your package manager. We used v19.1.0 for this setup. +- Ruby 3.0 +- Ruby gems: + - `bundler` + - `irb` + - `foreman` +- NodeJS v18 (LTS) +- NPM packages: + - `yarn` +- Postgres +- Redis -### Ruby +### macOS -Install ruby via your package manager of choice. +First, make sure you have Homebrew installed. Follow the instructions at +[brew.sh](https://brew.sh). -### Ruby Extras +Run the following to install all necessary packages: +``` +brew install ruby@3.0 foreman node yarn postgresql redis +``` -Your distro may package `bundle` and `irb` in separate packages (Arch does). If they are, install them too. -On Arch these are `ruby-bundler` and `ruby-irb`. +Ruby 3.0 is **keg-only** by default. Follow the instructions in the **Caveat** +to add it to your path. -### PostGreSQL +### Linux -Install postgres via your package manager. +We will assume that you know how to locate the correct packages for your distro. +That said, some distros package `bundler` and `irb` separately. Make sure that +you also install these. -### Redis +On Arch, you will need: -Install redis via your package manager. +- `ruby` +- `ruby-bundler` +- `ruby-irb` +- `ruby-foreman` -Enable and start the default `redis.service` +### Windows -## Setting up the Environment +Unfortunately, none of the authors use Windows. Contributions welcome! -In the following instructions, replace USER with your *nix user name. +## Database -1. Add yourself to the postgres group with `sudo usermod -a -G postgres USER`. You'll need to log out and back in to -update your groups. -2. Navigate to the root of this repo. -2. Set up a local DB cluster with `pg_ctl -D data/postgres15 initdb -o '-U mastodon --auth-host=trust'`. -3. Add the line `unix_socket_directories='.'` to `data/postgres15/postgresql.conf`. -4. Run the DB with `pg_ctl -D data/postgres15 start`. -5. Run `bundle config set --local path 'vendor/bundle`. This will store the all the ruby gems locally so that we can - avoid interfering with system config. -6. Run `bundle install`. -7. Run `yarn install`. -8. Run `bundle exec rake db:setup`. If this fails, you can use `bundle exec rake db:reset` to forcibly regenerate it. +In the root of this repository, go through the following script: +```sh +# Create a folder for local data +mkdir -p data + +# Set up a local database +pg_ctl -D data/postgres initdb -o '-U mastodon --auth-host=trust' + +# Use the data/postgres folder for the DB connection unix socket +# +# If you don't know what that means, it's just a way for Mastodon to communicate +# with a database on the same machine efficiently. +# +# See: https://manpages.ubuntu.com/manpages/jammy/man7/unix.7.html +echo 'unix_socket_directories = .' >> data/postgres/postgresql.conf + +# Start the database +pg_ctl -D data/postgres start --silent +``` + +## Redis + +In the root of this repository, run the following: +```sh +# Start Redis +redis-server ./redis-dev.conf + +# [Optional] Stop Redis +kill "$(cat ./data/redis/redis-dev.pid)" +``` + +## Ruby + +```sh +export RAILS_ENV=development + +# Bundle installs all Ruby gems globally by default, which might cause problems. +bundle config set --local path 'vendor/bundle' + +# [Apple Silicon] If using macOS on Apple Silicon, run the following: +# bundle config build.idn-ruby -- --with-idn-dir="$(brew --prefix libidn)" + +# Install dependencies using bundle (Ruby) and yarn (JS) +bundle install +yarn install + +# Setup the database using the pre-defined Rake task +# +# Rake is a command runner for Ruby projects. The `bundle exec` ensures that +# we use the version of Rake that this project requires. +bundle exec rake db:setup + +# [Optional] If that fails, run the following and try again: +# bundle exec rake db:reset +``` ## Running Mastodon -1. To make our lives easier, we'll use `foreman` to run the site, so use `gem install foreman`. -2. Run `export RAILS_ENV=development` and `export NODE_ENV=development`. +1. Run `export RAILS_ENV=development NODE_ENV=development`. a. Put these in your shell's .rc, or a script you can source if you want to skip this step in the future. -3. Run `bundle exec rake assets:precompile`. - a. If this explodes, complaining about `Hash`, you'll need to `export NODE_OPTIONS=--openssl-legacy-provider`. Same - deal as the above. - b. After doing this, you will need to `bundle exec rake assets:clobber` and then re-run +2. Run `bundle exec rake assets:precompile`. + a. If this explodes, complaining about `Hash`, you'll need to `export NODE_OPTIONS=--openssl-legacy-provider`. + b. After doing this, you will need to run `bundle exec rake assets:clobber` and then re-run `bundle exec rake assets:precompile`. -4. Run `foreman start` - +3. Run `foreman start` # Updates/Troubleshooting diff --git a/redis-dev.conf b/redis-dev.conf new file mode 100644 index 0000000000..706cd4afd1 --- /dev/null +++ b/redis-dev.conf @@ -0,0 +1,12 @@ +# This redis configuration is for development only + +daemonize yes + +dir ./data/redis +pidfile ./data/redis/redis-dev.pid + +unixsocket ./data/redis/redis-dev.sock +unixsocketperm 700 + +# Disable TCP +port 0 -- 2.41.0 From ea44d46ca980d2c304e1abce708c776633e4a881 Mon Sep 17 00:00:00 2001 From: Kouhai Date: Thu, 24 Nov 2022 18:58:23 -0600 Subject: [PATCH 14/18] update gitignore --- .gitignore | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/.gitignore b/.gitignore index 20f5fc0585..d045131724 100644 --- a/.gitignore +++ b/.gitignore @@ -5,9 +5,7 @@ # git config --global core.excludesfile '~/.gitignore_global' # Ignore local dotenv overrides -.env.development.local -.env.test.local -.env.production.local +.env.*.local # Ignore bundler config and downloaded libraries. /.bundle @@ -17,8 +15,8 @@ /db/*.sqlite3 /db/*.sqlite3-journal -# Ignore the dev postgres db we created. -/data/postgres*/* +# Ignore local data directory +/data # Ignore all logfiles and tempfiles. .eslintcache -- 2.41.0 From e60d7e3c43164951b562c3fd11a828fe5de3fd4e Mon Sep 17 00:00:00 2001 From: Rin Date: Fri, 25 Nov 2022 14:24:44 +1100 Subject: [PATCH 15/18] Add dir for redis data --- SETUP.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/SETUP.md b/SETUP.md index 1425e79e1e..d09de77b24 100644 --- a/SETUP.md +++ b/SETUP.md @@ -40,6 +40,8 @@ On Arch, you will need: - `ruby-bundler` - `ruby-irb` - `ruby-foreman` +- `redis` +- `postgresql` ### Windows @@ -71,6 +73,9 @@ pg_ctl -D data/postgres start --silent In the root of this repository, run the following: ```sh +# Create a folder for redis data +mkdir -p data/redis + # Start Redis redis-server ./redis-dev.conf -- 2.41.0 From bd0da3a4995b4a56c402bdd8cdb8a8b9173d50c6 Mon Sep 17 00:00:00 2001 From: Kouhai Date: Thu, 24 Nov 2022 23:01:03 -0600 Subject: [PATCH 16/18] fix redis conf --- redis-dev.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/redis-dev.conf b/redis-dev.conf index 706cd4afd1..ed197151d0 100644 --- a/redis-dev.conf +++ b/redis-dev.conf @@ -3,9 +3,9 @@ daemonize yes dir ./data/redis -pidfile ./data/redis/redis-dev.pid +pidfile ./redis-dev.pid -unixsocket ./data/redis/redis-dev.sock +unixsocket ./redis-dev.sock unixsocketperm 700 # Disable TCP -- 2.41.0 From c75ac19c8aa8a276539f21b97d1d593ebee15335 Mon Sep 17 00:00:00 2001 From: Kouhai Date: Fri, 25 Nov 2022 21:31:39 -0600 Subject: [PATCH 17/18] fix REDIS_URL unix socket path parsing for relative paths --- .env.development | 3 ++- .env.test | 2 +- lib/mastodon/redis_config.rb | 35 +++++++++++++++++++++++++++-------- streaming/index.js | 2 +- 4 files changed, 31 insertions(+), 11 deletions(-) diff --git a/.env.development b/.env.development index 1a11e918ef..0629bf94ce 100644 --- a/.env.development +++ b/.env.development @@ -1,6 +1,7 @@ LOCAL_DOMAIN=localhost +ALTERNATE_DOMAINS=mastodon.internal DB_HOST=$(pwd)/data/postgres DB_USER=mastodon DB_NAME=mastodon_dev -REDIS_URL=redis://$(pwd)/data/redis/redis-dev.sock +REDIS_URL=unix://./data/redis/redis-dev.sock diff --git a/.env.test b/.env.test index e3706868ca..1cfc25119a 100644 --- a/.env.test +++ b/.env.test @@ -7,4 +7,4 @@ LOCAL_HTTPS=true DB_HOST=$(pwd)/data/postgres DB_USER=mastodon DB_NAME=mastodon_dev -REDIS_URL=redis://$(pwd)/data/redis/redis-test.sock +REDIS_URL=unix://./data/redis/redis-dev.sock diff --git a/lib/mastodon/redis_config.rb b/lib/mastodon/redis_config.rb index 98dc4788d1..11ae63c7a8 100644 --- a/lib/mastodon/redis_config.rb +++ b/lib/mastodon/redis_config.rb @@ -1,17 +1,36 @@ # frozen_string_literal: true +require 'pathname' + def setup_redis_env_url(prefix = nil, defaults = true) - prefix = prefix.to_s.upcase + '_' unless prefix.nil? + prefix = "#{prefix.to_s.upcase}_" unless prefix.nil? prefix = '' if prefix.nil? + redis_url_key = "#{prefix}REDIS_URL" - return if ENV[prefix + 'REDIS_URL'].present? + if ENV[redis_url_key].present? + conn = +ENV["#{prefix}REDIS_URL"].sub(/redis:\/\//i, '') - password = ENV.fetch(prefix + 'REDIS_PASSWORD') { '' if defaults } - host = ENV.fetch(prefix + 'REDIS_HOST') { 'localhost' if defaults } - port = ENV.fetch(prefix + 'REDIS_PORT') { 6379 if defaults } - db = ENV.fetch(prefix + 'REDIS_DB') { 0 if defaults } + # Strip any prefixing `unix://` + unix = !conn.sub!(/^unix:\/\//i, '').nil? + # Strip any prefixing `./` + unix |= conn.sub!(/^(\.\/)+/, '') + unix |= conn.start_with?('/') - ENV[prefix + 'REDIS_URL'] = begin + if unix + pn = Pathname.new(conn) + pn = Pathname.getwd / pn if pn.relative? + ENV[redis_url_key] = "unix://#{pn}" + end + + return + end + + password = ENV.fetch("#{prefix}REDIS_PASSWORD") { '' if defaults } + host = ENV.fetch("#{prefix}REDIS_HOST") { 'localhost' if defaults } + port = ENV.fetch("#{prefix}REDIS_PORT") { 6379 if defaults } + db = ENV.fetch("#{prefix}REDIS_DB") { 0 if defaults } + + ENV["#{prefix}REDIS_URL"] = begin if [password, host, port, db].all?(&:nil?) ENV['REDIS_URL'] else @@ -27,7 +46,7 @@ setup_redis_env_url(:cache, false) setup_redis_env_url(:sidekiq, false) namespace = ENV.fetch('REDIS_NAMESPACE', nil) -cache_namespace = namespace ? namespace + '_cache' : 'cache' +cache_namespace = namespace ? "#{namespace}_cache" : 'cache' sidekiq_namespace = namespace REDIS_CACHE_PARAMS = { diff --git a/streaming/index.js b/streaming/index.js index cecb53ed5b..b09e7e6b6d 100644 --- a/streaming/index.js +++ b/streaming/index.js @@ -18,7 +18,7 @@ const env = process.env.NODE_ENV || 'development'; const alwaysRequireAuth = process.env.LIMITED_FEDERATION_MODE === 'true' || process.env.WHITELIST_MODE === 'true' || process.env.AUTHORIZED_FETCH === 'true'; dotenv.config({ - path: env === 'production' ? '.env.production' : '.env', + path: env === 'production' || env === 'development' ? `.env.${env}` : '.env', }); log.level = process.env.LOG_LEVEL || 'verbose'; -- 2.41.0 From 41f55e5d8f5b532d26bb47619177bac26c8322cf Mon Sep 17 00:00:00 2001 From: Kouhai Date: Fri, 25 Nov 2022 22:03:05 -0600 Subject: [PATCH 18/18] update upstream glitch README/CONTRIBUTING --- CONTRIBUTING.md | 17 +++++++++++------ README.md | 24 +++++++++++++++--------- 2 files changed, 26 insertions(+), 15 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 049cd4e29b..b4454309e9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,6 +1,6 @@ -# Contributing to Mastodon Glitch Edition # +# Contributing to Mastodon Glitch+Treehouse Edition # -Thank you for your interest in contributing to the `glitch-soc` project! +Thank you for your interest in contributing to the **Treehouse Mastodon** project! Here are some guidelines, and ways you can help. > (This document is a bit of a work-in-progress, so please bear with us. @@ -8,14 +8,19 @@ Here are some guidelines, and ways you can help. ## Planning ## -Right now a lot of the planning for this project takes place in our development Discord, or through GitHub Issues and Projects. +Right now a lot of the planning for this project takes place in the `#fediverse` +channel of the Treehouse Discord, or through Gitea Issues. + We're working on ways to improve the planning structure and better solicit feedback, and if you feel like you can help in this respect, feel free to give us a holler. ## Documentation ## -The documentation for this repository is available at [`glitch-soc/docs`](https://github.com/glitch-soc/docs) (online at [glitch-soc.github.io/docs/](https://glitch-soc.github.io/docs/)). -Right now, we've mostly focused on the features that make this fork different from upstream in some manner. -Adding screenshots, improving descriptions, and so forth are all ways to help contribute to the project even if you don't know any code. +The upstream Glitch documentation for this repository is available at [`glitch-soc/docs`](https://github.com/glitch-soc/docs) (online at [glitch-soc.github.io/docs/](https://glitch-soc.github.io/docs/)). + +## Setup ## + +For a some-batteries-required guide to setting up a development environment for this repository, read Rin's excellent +[SETUP.md](https://gitea.treehouse.systems/treehouse/mastodon/src/branch/main/SETUP.md). ## Frontend Development ## diff --git a/README.md b/README.md index 256f2d2def..c04e743072 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,20 @@ -# Mastodon Glitch Edition # +# Mastodon Glitch+Treehouse Edition # -> Now with automated deploys! +> Now with bunny ears! -[![Build Status](https://img.shields.io/circleci/project/github/glitch-soc/mastodon.svg)][circleci] -[![Code Climate](https://img.shields.io/codeclimate/maintainability/glitch-soc/mastodon.svg)][code_climate] +So here's the deal: we all work on this code, and anyone who uses that does so absolutely at their own risk. Can you dig it? -[circleci]: https://circleci.com/gh/glitch-soc/mastodon -[code_climate]: https://codeclimate.com/github/glitch-soc/mastodon +Specifically, this fork-of-a-fork is intended for Treehouse use only. Unless +otherwise communicated, we will not put effort into supporting other deployments +or upstreaming our patches. -So here's the deal: we all work on this code, and anyone who uses that does so absolutely at their own risk. can you dig it? +## Links -- You can view documentation for this project at [glitch-soc.github.io/docs/](https://glitch-soc.github.io/docs/). -- And contributing guidelines are available [here](CONTRIBUTING.md) and [here](https://glitch-soc.github.io/docs/contributing/). +- You can view upstream Glitch's documentation for this project at [glitch-soc.github.io/docs/](https://glitch-soc.github.io/docs/). +- Contributing guidelines are available [here](CONTRIBUTING.md). + +## Known Deployments + +- Treehouse Social: [social.treehouse.systems](https://social.treehouse.systems) +- VT Social: [vt.social](https://vt.social) +- Unstable Systems: [unstable.systems](https://unstable.systems) -- 2.41.0