diff --git a/analytics/README.md b/analytics/README.md index 86607cf..df53a12 100644 --- a/analytics/README.md +++ b/analytics/README.md @@ -1,2 +1,5 @@ Placeholder for: -https://github.com/dwyl/learn-devops/issues/91 \ No newline at end of file +https://github.com/dwyl/learn-devops/issues/91 + +Meanwhile, see: +https://github.com/dwyl/learn-analytics/tree/main/plausible \ No newline at end of file diff --git a/cheatsheet.md b/cheatsheet.md new file mode 100644 index 0000000..7b54364 --- /dev/null +++ b/cheatsheet.md @@ -0,0 +1,12 @@ +# Cheat Sheet + +This [cheat sheet](https://en.wikipedia.org/wiki/Cheat_sheet) +helps us manage our `DevOps` _fast_. + +## Update & Reboot Ubuntu Server + +```sh +sudo apt update -y && sudo apt full-upgrade -y && sudo apt autoremove -y && sudo apt clean -y && sudo apt autoclean -y && sudo reboot +``` + +The `alias` for this command on our servers is `upr`. \ No newline at end of file diff --git a/hetzner/README.md b/hetzner/README.md new file mode 100644 index 0000000..27bd7be --- /dev/null +++ b/hetzner/README.md @@ -0,0 +1,56 @@ +
+ +![hetzner-logo-banner](https://github.com/user-attachments/assets/5b3b5a63-33ae-41d6-bcff-6a25744db465 "Hetzner Logo") + +We've made the switch to `Hetzner`, +we think you should too. + +
+ +# Why? + +After more than a decade using various "Cloud" providers +including all the _Big_ Tech (`AWS`, `Azure`, `GCP`, `DigitalOcean`, `Linode`) +and many PaaS such as Lambda, Fly.io, Heroku, Vercel, etc. +we've finally bitten the bullet and gone _back_ to our roots; Servers! + +## Brief Aside on Motivation + +There two types of motivation: towards and away. +**Toward** is the **_positive_** motivation such as getting fit/healthy. +**Away** motivation is when we want to _avoid_ something, like being unfit. +Both types of motivation have their place. +Some people are exclusively motivated by loss/pain/risk aversion, +while others are driven by gain/reward/returns and downplay the downside. +I go through phases of being super risk averse +and others when I'm happy to take calculated risks +that others who haven't done the math think are _crazy_. + +In the case of self-hosting our web apps on **barebones servers**, +we have _plenty_ of experience from the _pre-AWS_ days. +Yes, this ages us, but the experience was formative. +And means I'm not afraid to dive in. + +I'm motivated _away_ from the +[data loss](https://github.com/dwyl/auth/issues/325#issuecomment-1792297886) +we experienced on `Fly.io` +and _toward_ the high availability/affordability of `Hetzner`. +I know this will require some setup/config work, +but am undeterred; +that's why we write systematic & meticulous notes! + +> “_Notes aren’t a **record** of my thinking process. +> They **are** my thinking process_.” +~ Richard Feynman + + +## Recommended Reading + ++ Good summary including "incidents": +[wikipedia.org/wiki/Hetzner](https://en.wikipedia.org/wiki/Hetzner) ++ `Hetzner` "about" page: +[hetzner.com/unternehmen/ueber-uns](https://www.hetzner.com/unternehmen/ueber-uns/) ++ Sustainability report: +[hetzner.com/unternehmen/nachhaltigkeit](https://www.hetzner.com/unternehmen/nachhaltigkeit) ++ Finances: +[northdata.com/Hetzner](https://www.northdata.com/Hetzner%20Online%20GmbH,%20Gunzenhausen/Amtsgericht%20Ansbach%20HRB%206089) \ No newline at end of file diff --git a/nginx/README.md b/nginx/README.md new file mode 100644 index 0000000..1b3f026 --- /dev/null +++ b/nginx/README.md @@ -0,0 +1,319 @@ +
+ +# `nginx` _Speedy_ Setup + +This is a speed run of using `nginx` +to proxy an app running on a `Hetzner` server. + +
+ +## 1. Install `nginx` on `Ubuntu` + +`SSH` into the virtual machine, e.g: + +```sh +ssh root@88.99.81.115 +``` + +Ensure that everything is up-to-date on the VM: + +```sh +sudo apt update -y && sudo apt full-upgrade -y && sudo apt autoremove -y && sudo apt clean -y && sudo apt autoclean -y +``` + +Followed by: + +```sh +sudo reboot +``` + +Now we can proceed with installing `nginx`. + +Official instructions: +https://ubuntu.com/tutorials/install-and-configure-nginx#1-overview + +```sh +sudo apt install nginx +``` + +That installs and automatically starts the `nginx` server. + +Check the status: + +```sh +service nginx status +``` + +Output: + +```sh +● nginx.service - A high performance web server and a reverse proxy server + Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled) + Active: active (running) since Fri 2025-03-21 10:49:46 UTC; 56s ago + Docs: man:nginx(8) + Process: 754 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) + Process: 778 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) + Main PID: 803 (nginx) + Tasks: 3 (limit: 4540) + Memory: 3.7M (peak: 3.8M) + CPU: 34ms + CGroup: /system.slice/nginx.service + ├─803 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;" + ├─804 "nginx: worker process" + └─805 "nginx: worker process" +``` + +Visit: +http://88.99.81.115 + +![nginx-running](https://github.com/user-attachments/assets/f8754c78-7243-4844-9ab6-eb642d4ab2e7) + + +## 2. Certbot + +Instructions: +https://certbot.eff.org/instructions?ws=nginx&os=ubuntufocal + +```sh +sudo snap install --classic certbot +``` + +Output: + +```sh +2025-03-21T11:48:11Z INFO Waiting for automatic snapd restart... +certbot 3.3.0 from Certbot Project (certbot-eff✓) installed +``` + +Link the command: + +```sh +sudo ln -s /snap/bin/certbot /usr/bin/certbot +``` + +Basic `certbot` setup for an `nginx` server: + +```sh +sudo certbot --nginx +``` + +Output: + +```sh +Requesting a certificate for dwy.is + +Successfully received certificate. +Certificate is saved at: /etc/letsencrypt/live/dwy.is/fullchain.pem +Key is saved at: /etc/letsencrypt/live/dwy.is/privkey.pem +This certificate expires on 2025-06-19. +These files will be updated when the certificate renews. +Certbot has set up a scheduled task to automatically renew this certificate in the background. + +Deploying certificate +Successfully deployed certificate for dwy.is to /etc/nginx/sites-enabled/default +Congratulations! You have successfully enabled HTTPS on https://dwy.is + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +If you like Certbot, please consider supporting our work by: + * Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate + * Donating to EFF: https://eff.org/donate-le +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +``` + +Dry run renewal: + +```sh +sudo certbot renew --dry-run +``` + +```sh +Saving debug log to /var/log/letsencrypt/letsencrypt.log + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +Processing /etc/letsencrypt/renewal/dwy.is.conf +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +Account registered. +Simulating renewal of an existing certificate for dwy.is + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +Congratulations, all simulated renewals succeeded: + /etc/letsencrypt/live/dwy.is/fullchain.pem (success) +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +``` + +The full config including the TLS is in +`/etc/nginx/sites-available/default` + +Now create a new config file +_just_ for the subdomain. + +## 3. Configure `nginx` Subdomain + +```sh +vi /etc/nginx/sites-enabled/autobase +``` + +Paste the contents from this file: +`nginx/sites-enabled/autobase` + +Test `nginx` config: + +```sh +nginx -t +``` + +You should see output similar to the following: + +```sh +nginx: the configuration file /etc/nginx/nginx.conf syntax is ok +nginx: configuration file /etc/nginx/nginx.conf test is successful +``` + +Test a specific configuration file: + +```sh +nginx -t -c /path/to/conf +``` + +In our case: + +```sh +nginx -t -c /etc/nginx/sites-enabled/autobase +``` + +If your config fails the test for any reason, +try checking it online: +[google.com/search?q=nginx+syntax+check+online](https://www.google.com/search?q=nginx+syntax+check+online) +e.g: +[getpagespeed.com/check-nginx-config](https://www.getpagespeed.com/check-nginx-config) + +Restart `nginx`: + +```sh +sudo service nginx restart +``` + +## 4. Wildcard Certificate + +In our case, I actually wanted a wildcard certificate +so that I can add any subdomain I want later. + +Wildcard Certificate instructions: +https://www.baeldung.com/linux/letsencrypt-certbot-add-subdomains + +Sample command: + +```sh +sudo certbot certonly -i nginx -d example.com -d *.example.com +``` + +In our case: + +```sh +sudo certbot certonly -i nginx -d dwy.is -d *.dwy.is -v +``` + +Output: + +```sh +Saving debug log to /var/log/letsencrypt/letsencrypt.log +Plugins selected: Authenticator manual, Installer None + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +You have an existing certificate that contains a portion of the domains you +requested (ref: /etc/letsencrypt/renewal/dwy.is.conf) + +It contains these names: dwy.is + +You requested these names for the new certificate: dwy.is, *.dwy.is. + +Do you want to expand and replace this existing certificate with the new +certificate? +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +(E)xpand/(C)ancel: E +Renewing an existing certificate for dwy.is and *.dwy.is +Performing the following challenges: +dns-01 challenge for dwy.is + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +Please deploy a DNS TXT record under the name: + +_acme-challenge.dwy.is. + +with the following value: + +8GC-85xs1BGQDlU7YKpxA5fyHBV20PqBU8aMA9lAN10 + +Before continuing, verify the TXT record has been deployed. Depending on the DNS +provider, this may take some time, from a few seconds to multiple minutes. You can +check if it has finished deploying with aid of online tools, such as the Google +Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.dwy.is. +Look for one or more bolded line(s) below the line ';ANSWER'. It should show the +value(s) you've just added. + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +Press Enter to Continue +``` + +I created the `TXT` record: + +https://ap.www.namecheap.com/domains/domaincontrolpanel/dwy.is/advancedns + +![dwyis-txt-record](https://github.com/user-attachments/assets/80ddea19-d06c-4d71-8c8e-f86ee2acd9dc) + +But this was incorrect! +The host needed to be `_acme-challenge` +***NOT*** `_acme-challenge.dwy.is` +as was implied by `certbot`. +i.e. the domain `dwy.is` should not be in the host! + +https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.dwy.is + +![google-dig-txt](https://github.com/user-attachments/assets/6ccbb156-6c34-4d67-9c13-f69db9b47a76) + +I refreshed this like a million times over `48h` +but it never updated. + +```sh +dig -t txt _acme-challenge.dwy.is +``` + +I decided to contact `NameCheap` support via live chat: +https://www.namecheap.com/help-center/live-chat + +They were helpful and together we determined that _I_ had misconfigured the `TXT` record ... 🤦 + +Updated config: + +https://ap.www.namecheap.com/domains/domaincontrolpanel/dwy.is/advancedns + +![dwy.is-dns-txt-record](https://github.com/user-attachments/assets/c19e6cae-132c-4f70-ba99-6bd8829f0d13) + +Full transcript: [Chat_Transcript_23_Mar_2025.pdf](https://github.com/user-attachments/files/19410954/Chat_Transcript_23_Mar_2025.pdf) + +Final output: + +```sh +Renewing an existing certificate for dwy.is and *.dwy.is +Reloading nginx server after certificate issuance + +Successfully received certificate. +Certificate is saved at: /etc/letsencrypt/live/dwy.is/fullchain.pem +Key is saved at: /etc/letsencrypt/live/dwy.is/privkey.pem +This certificate expires on 2025-06-21. +These files will be updated when the certificate renews. +Certbot has set up a scheduled task to automatically renew this certificate in the background. + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +If you like Certbot, please consider supporting our work by: + * Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate + * Donating to EFF: https://eff.org/donate-le +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +``` + +Working! + +![autobase.dwy.is-with-ssl](https://github.com/user-attachments/assets/15411040-860f-4a56-9c2d-91fc8702c318) + +Also used: +https://dnschecker.org/#TXT/_acme-challenge.dwy.is \ No newline at end of file diff --git a/nginx/nginx.conf b/nginx/nginx.conf new file mode 100644 index 0000000..0388fd0 --- /dev/null +++ b/nginx/nginx.conf @@ -0,0 +1,84 @@ +# /etc/nginx/nginx.conf +user www-data; +worker_processes auto; +pid /run/nginx.pid; +error_log /var/log/nginx/error.log; +include /etc/nginx/modules-enabled/*.conf; + +events { + worker_connections 768; + # multi_accept on; +} + +http { + + ## + # Basic Settings + ## + + sendfile on; + tcp_nopush on; + types_hash_max_size 2048; + # server_tokens off; + + # server_names_hash_bucket_size 64; + # server_name_in_redirect off; + + include /etc/nginx/mime.types; + default_type application/octet-stream; + + ## + # SSL Settings + ## + + ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE + ssl_prefer_server_ciphers on; + + ## + # Logging Settings + ## + + access_log /var/log/nginx/access.log; + + ## + # Gzip Settings + ## + + gzip on; + + # gzip_vary on; + # gzip_proxied any; + # gzip_comp_level 6; + # gzip_buffers 16 8k; + # gzip_http_version 1.1; + # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript; + + ## + # Virtual Host Configs + ## + + include /etc/nginx/conf.d/*.conf; + include /etc/nginx/sites-enabled/*; +} + + +#mail { +# # See sample authentication script at: +# # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript +# +# # auth_http localhost/auth.php; +# # pop3_capabilities "TOP" "USER"; +# # imap_capabilities "IMAP4rev1" "UIDPLUS"; +# +# server { +# listen localhost:110; +# protocol pop3; +# proxy on; +# } +# +# server { +# listen localhost:143; +# protocol imap; +# proxy on; +# } +#} \ No newline at end of file diff --git a/nginx/sites-enabled/autobase b/nginx/sites-enabled/autobase new file mode 100644 index 0000000..057c1cd --- /dev/null +++ b/nginx/sites-enabled/autobase @@ -0,0 +1,43 @@ +server { + listen 80; + server_name autobase.dwy.is; + + return 301 https://$host$request_uri; +} + +server { + server_name autobase.dwy.is; + + location / { + # stackoverflow.com/questions/14501047/add-response-header-nginx-proxy-pass + # 1. hide the Access-Control-Allow-Origin from the server response + proxy_hide_header Access-Control-Allow-Origin; + # 2. add a new custom header that allows all * origins instead + add_header 'Access-Control-Allow-Origin' '*'; + add_header 'Access-Control-Allow-Credentials' 'true'; + add_header 'Access-Control-Allow-Methods' '*'; + add_header 'Access-Control-Allow-Headers' '*'; + + proxy_pass http://172.17.0.1:82; + } + + listen 443 ssl; # managed by Certbot + ssl_certificate /etc/letsencrypt/live/dwy.is/fullchain.pem; # managed by Certbot + ssl_certificate_key /etc/letsencrypt/live/dwy.is/privkey.pem; # managed by Certbot + include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot +} + +server { + listen 8080 ssl; + ssl_certificate /etc/letsencrypt/live/dwy.is/fullchain.pem; # managed by Certbot + ssl_certificate_key /etc/letsencrypt/live/dwy.is/privkey.pem; # managed by Certbot + include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot + + server_name autobase.dwy.is; + + location / { + proxy_pass http://172.17.0.1:8082/; + } +} \ No newline at end of file diff --git a/nginx/sites-enabled/default b/nginx/sites-enabled/default new file mode 100644 index 0000000..ebc6122 --- /dev/null +++ b/nginx/sites-enabled/default @@ -0,0 +1,161 @@ +## +# You should look at the following URL's in order to grasp a solid understanding +# of Nginx configuration files in order to fully unleash the power of Nginx. +# https://www.nginx.com/resources/wiki/start/ +# https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/ +# https://wiki.debian.org/Nginx/DirectoryStructure +# +# In most cases, administrators will remove this file from sites-enabled/ and +# leave it as reference inside of sites-available where it will continue to be +# updated by the nginx packaging team. +# +# This file will automatically load configuration files provided by other +# applications, such as Drupal or Wordpress. These applications will be made +# available underneath a path with that package name, such as /drupal8. +# +# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples. +## + +# Default server configuration +# +server { + listen 80 default_server; + listen [::]:80 default_server; + + # SSL configuration + # + # listen 443 ssl default_server; + # listen [::]:443 ssl default_server; + # + # Note: You should disable gzip for SSL traffic. + # See: https://bugs.debian.org/773332 + # + # Read up on ssl_ciphers to ensure a secure configuration. + # See: https://bugs.debian.org/765782 + # + # Self signed certs generated by the ssl-cert package + # Don't use them in a production server! + # + # include snippets/snakeoil.conf; + + root /var/www/html; + + # Add index.php to the list if you are using PHP + index index.html index.htm index.nginx-debian.html; + + server_name _; + + location / { + # First attempt to serve request as file, then + # as directory, then fall back to displaying a 404. + try_files $uri $uri/ =404; + } + + # pass PHP scripts to FastCGI server + # + #location ~ \.php$ { + # include snippets/fastcgi-php.conf; + # + # # With php-fpm (or other unix sockets): + # fastcgi_pass unix:/run/php/php7.4-fpm.sock; + # # With php-cgi (or other tcp sockets): + # fastcgi_pass 127.0.0.1:9000; + #} + + # deny access to .htaccess files, if Apache's document root + # concurs with nginx's one + # + #location ~ /\.ht { + # deny all; + #} +} + + +# Virtual Host configuration for example.com +# +# You can move that to a different file under sites-available/ and symlink that +# to sites-enabled/ to enable it. +# +#server { +# listen 80; +# listen [::]:80; +# +# server_name example.com; +# +# root /var/www/example.com; +# index index.html; +# +# location / { +# try_files $uri $uri/ =404; +# } +#} + +server { + + # SSL configuration + # + # listen 443 ssl default_server; + # listen [::]:443 ssl default_server; + # + # Note: You should disable gzip for SSL traffic. + # See: https://bugs.debian.org/773332 + # + # Read up on ssl_ciphers to ensure a secure configuration. + # See: https://bugs.debian.org/765782 + # + # Self signed certs generated by the ssl-cert package + # Don't use them in a production server! + # + # include snippets/snakeoil.conf; + + root /var/www/html; + + # Add index.php to the list if you are using PHP + index index.html index.htm index.nginx-debian.html; + server_name dwy.is; # managed by Certbot + + + location / { + # First attempt to serve request as file, then + # as directory, then fall back to displaying a 404. + try_files $uri $uri/ =404; + } + + # pass PHP scripts to FastCGI server + # + #location ~ \.php$ { + # include snippets/fastcgi-php.conf; + # + # # With php-fpm (or other unix sockets): + # fastcgi_pass unix:/run/php/php7.4-fpm.sock; + # # With php-cgi (or other tcp sockets): + # fastcgi_pass 127.0.0.1:9000; + #} + + # deny access to .htaccess files, if Apache's document root + # concurs with nginx's one + # + #location ~ /\.ht { + # deny all; + #} + + + listen [::]:443 ssl ipv6only=on; # managed by Certbot + listen 443 ssl; # managed by Certbot + ssl_certificate /etc/letsencrypt/live/dwy.is/fullchain.pem; # managed by Certbot + ssl_certificate_key /etc/letsencrypt/live/dwy.is/privkey.pem; # managed by Certbot + include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot + +} +server { + if ($host = dwy.is) { + return 301 https://$host$request_uri; + } # managed by Certbot + + + listen 80 ; + listen [::]:80 ; + server_name dwy.is; + return 404; # managed by Certbot +} \ No newline at end of file diff --git a/postgres/README.md b/postgres/README.md index cc360fa..de9290e 100644 --- a/postgres/README.md +++ b/postgres/README.md @@ -1,23 +1,26 @@
-Postgres logo wide # Deployment
-`Postgres` deployment is divided into two options: +`Postgres` deployment is split into two options: 1. Managed - the infrastructure provider manages the instances for you including data backups and failover in the event of a crash/corruption. + 2. Unmanaged - we the engineers or operations team need to manage it including data integrity and availability. Both have their place. But if you are not an _experienced_ -Database Administrator (DBA) -or System Administrator (SysAdmin), +Database Administrator +([DBA](https://en.wikipedia.org/wiki/Database_administration)) +or System Administrator +([SysAdmin](https://en.wikipedia.org/wiki/System_administrator)), you should seriously consider a _managed_ service. The risk of data loss greatly outweighs the cost of a _managed_ service. @@ -35,10 +38,12 @@ There are _many_ other options for "cloud" providers for managed `Postgres` and other `SQL` databases. We looked at all the major ones including `AWS`, `GCP`, `Azure`. -Sadly, `AWS Aurara` while appealing, -has _deliberately_ confusing pricing: -https://aws.amazon.com/rds/aurora/pricing +Sadly, `AWS Aurara` while appealing, +has **_deliberately_ confusing pricing**: +[aws.amazon.com/rds/aurora/pricing](https://aws.amazon.com/rds/aurora/pricing) They have costs for I/O requests, storage, backtrack (backup) and data transfer. By contrast `DigitalOcean` has _transparent_ pricing based on the VPS (Memory, CPU and SSD) used. +But `DigitalOcean` also gets _very_ expensive ... +So we decided to invest the time to _self-manage_. diff --git a/postgres/autobase-ha-cluster.md b/postgres/autobase-ha-cluster.md new file mode 100644 index 0000000..57ede7c --- /dev/null +++ b/postgres/autobase-ha-cluster.md @@ -0,0 +1,387 @@ +
+ +# High Availability `Postgres` Cluster With `Autobase` + +![autobase-hero-image](https://github.com/user-attachments/assets/51ff8785-2f07-4a23-8892-41ff6a6d2aaa) + +
+ + +Deploy a +[high availability](https://en.wikipedia.org/wiki/High_availability) +`Postgres` database cluster +(on +[`Hetzner`](../hetzner)) +using [**`autobase`**](https://autobase.tech). + + + +Along the way we will clarify as many of the steps as possible.
+But please keep in mind it's _not possible_ +to cover everything in a **7 minute video**. + +If you have questions, suggestions or just want to say hi, +**please comment on YouTube**; +thanks. + +With all that out of the way, lets dive in! + +## 1. Login to `Hetzner` Cloud + +When you _first_ login to `Hetzner`, +you will see the message: + +"You don't have any servers yet." + +hetzner-no-servers + +Click the "**Add Server**" button to begin your quest! + +> If you don't yet have a `Hezner` account, +> please consider using our **referral link**: +> [hetzner.cloud/?ref=ahpZuUB3t0XI](https://hetzner.cloud/?ref=ahpZuUB3t0XI) 🔗 +> to get **`$20`** in credit. 💵
+> Helps us do what we love too. Thanks. ❤ + +## 2. Create a New "Cloud" Virtual Private Server (VPS) + +Select all the default options, +add your `ssh` `public` key +and create your server. + +new-server-created + +## 3. `SSH` into the `Hetzner` Server + +Use your `Terminal` to login to the newly created `Hetzner`server, e.g: + +```sh +ssh root@116.202.31.52 +``` + +Once you have successfully connected via `ssh`, +a best practice we recommend is to run a quick update. + +### Update The Server + +Run the following command chain: + +```sh +sudo apt update -y && sudo apt full-upgrade -y && sudo apt autoremove -y && sudo apt clean -y && sudo apt autoclean -y +``` + +> Updates and installs usually take a couple of minutes. +> We speed installs up for brevity. + +With everything up-to-date, install the necessary dependencies. + + +## 4. Install Dependencies + +As per the `autobase` getting started guide: +[autobase.tech/docs#getting-started](https://autobase.tech/docs#getting-started) +run the following command to get the necessary dependencies: + +```sh +sudo apt update && sudo apt install -y python3-pip sshpass git +pip3 install ansible +``` + +### Install `Docker` + +Install `Docker` on the `Ubuntu` server +following the installation instructions +in the **official `Docker` docs**: +https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository + +Run: + +```sh +# Add Docker's official GPG key: +sudo apt-get update +sudo apt-get install ca-certificates curl +sudo install -m 0755 -d /etc/apt/keyrings +sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc +sudo chmod a+r /etc/apt/keyrings/docker.asc + +# Add the repository to Apt sources: +echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ + $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ + sudo tee /etc/apt/sources.list.d/docker.list > /dev/null +sudo apt-get update +``` + +Followed by: + +```sh +sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin +``` + +Verify that the installation is successful by running the `hello-world` image: + +```sh +$ sudo docker run hello-world +``` + +With that confirmed working, +go back to the previous step and run the `autobase`command. + +## 5. Run `autobase` Console Boot Script + +Install and run the `latest` version of `Autobase` console. + +Sample: + +```sh +docker run -d --name autobase-console \ + --publish 80:80 \ + --publish 8080:8080 \ + --env PG_CONSOLE_API_URL=http://localhost:8080/api/v1 \ + --env PG_CONSOLE_AUTHORIZATION_TOKEN=secret_token \ + --env PG_CONSOLE_DOCKER_IMAGE=autobase/automation:latest \ + --volume console_postgres:/var/lib/postgresql \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + --volume /tmp/ansible:/tmp/ansible \ + --restart=unless-stopped \ + autobase/console:latest +``` + +You will nee to replace the `localhost` in the `PG_CONSOLE_API_URL` +with the IP (v4) address of your server +and `secret_token` for the `PG_CONSOLE_AUTHORIZATION_TOKEN` + ++ IP: 116.202.31.52 (yours will be different!) ++ Token: 5b0b6259-a7d4-4435-947dba (create your own!) + +Actual: + +```sh +docker run -d --name autobase-console \ + --publish 80:80 \ + --publish 8080:8080 \ + --env PG_CONSOLE_API_URL=http://116.202.31.52:8080/api/v1 \ + --env PG_CONSOLE_AUTHORIZATION_TOKEN=5b0b6259-a7d4-4435-947dba \ + --env PG_CONSOLE_DOCKER_IMAGE=autobase/automation:latest \ + --volume console_postgres:/var/lib/postgresql \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + --volume /tmp/ansible:/tmp/ansible \ + --restart=unless-stopped \ + autobase/console:latest +``` + +Confirm it worked with the `docker ps` command. +You should see something similar to the following: + +```sh +CONTAINER ID   IMAGE                     COMMAND                  CREATED              STATUS              PORTS                                                                                    NAMES + +9740dfd66c42   autobase/console:latest   "/usr/bin/supervisor…"   About a minute ago   Up About a minute   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 5432/tcp   autobase-console +``` + +## 6. Login To `autobase` Console Web UI + +Visit the IP Address of your server in you web browser e.g: +http://116.202.31.52 + +You should see a login screen: + +autobase-console-login + +Copy-paste the Token you defined in step 5 above. + +When you first login you should see that there are **No Postgres Clusters**: + +autobase-no-clusters + +## 7. Create Postgres Cluster + +Click the "**CREATE CLUSTER**" button: + +create-cluster-button + +Select `hetzner`and the datacenter region you prefer, in our case Europe: + +create-cluster-europe + +The default disk storage is **`100Gb`**; + +cluster-disk-storage + +this is _way_ too much for most simple projects. +lower it to **`10Gb`** for each instance to instantly save **50%** of the cost! +(you're welcome!) + +disk-storage-10gb + +> **Note**: all values for `DISK`storage, `RAM`, and `CPU`can easily be scaled later. + +Finally, you'll need to add your `public` SSH key. + +### Copy Your SSH Public Key + +```sh +cat ~/.ssh/id_ed25519.pub | pbcopy +``` + +Paste it into the **`SSH public key`** field: + +add-ssh-key + +Then scroll down and click the "**CREATE CLUSTER**" button. + +You will see a modal window appear prompting you to input a `Hetzner`API Key: + +hetzner-api-key-modal + +### 8. Generate an API token + +Follow the instructions in the official `Hetzner` docs: +https://docs.hetzner.com/cloud/api/getting-started/generating-api-token/ + +In the `Hetzner`console, navigate to **Security** > **API tokens**. +You should see the message: +"**You haven't generated an API token yet.**" + +hetzner-api-tokens + +Click on the "**Generate API token**" button: + +generate-api-token + +That will open _another_ modal window, +input the description for your key, +e.g: +"postgres-cluster-api-key" +and select "**Read and Write**": + +generate-api-token-modal + +Finally, click on the "**Generate API token**" button. +You should see a confirmation message: + +token-created + +Click to reveal the token you created: + +copy-token + +Copy the token to your clipboard, e.g: + +```sh +zH2qdgCeogrKjVKgV7sngMRxCfewgSdDARUBr8yqcjuHhGzlNdY72H13Sjh1il2D +``` + +> **Note**: for security reasons, this API key is no longer valid. + +Paste it into the Cluster creation window: + +paste-token-in-auto-window + +_Optionally_ save the API Key to the console and then +click "**CREATE CLUSTER**": + +create-cluster + +created: + +cluster-created + +Cluster details: + +cluster-details + +The `Postgres` cluster _appears_ to be deployed, +but how do we _know_ that it worked? + +## 9. Test The Cluster! 👩‍🔬 + +First: _connect_ to the **primary** `Postgres` instance. +In our case this is: `10.0.1.4` + +postgres-primary + +Sample: + +```sh +export PGPASSWORD='password'; +psql -h 127.0.0.1 -p 5432 -U postgres -d postgres +``` + +Get the **Password** and **Port** from the **Connection info** panel: + +postgres-connection-info + +Actual: + +```sh +export PGPASSWORD='9Djw2LNRMWwaDS1F9TlxeXiGj4dV3zNk'; +psql -h 88.99.81.115 -p 5432 -U postgres -d postgres +``` + +```sh +psql -h 10.0.1.4 -p 6432 -U postgres -d postgres +``` + +```sh +psql -h 10.0.1.4 -p 6432 -U postgres -d postgres -c "select version()" +``` + +Got the following error: + +```sh +Command 'psql' not found, but can be installed with: + +apt install postgresql-client-common +``` + +This is a barebones `Ubuntu` instance, remember, so it's not surprising that it doesn't have `psql` installed. So follow the instruction and install it: + +```sh +sudo apt install postgresql-client-common +``` + +The output is: + +But when trying to run `psql` again, we still get an error: + +```sh +Error: You must install at least one postgresql-client- package +``` + +## Outro: + +Given that this is a technical guide for an evolving system, +it may need to be enhanced/extended or updated in future, +that will be done on GitHub; +_everyone_ is welcome to and _encouraged_ to contribute! +Again, link in the description. + +Thanks for watching/listening. +If you found it useful and want to see more, +please subscribe. + +## Privacy Disclaimer + +By the time you read/watch this, +all of the sensitive data such as passwords, IP addresses, +public keys and auth tokens will have been updated. +This avoids anyone getting ideas about accessing backend systems. + +We publish our notes and videos on how we do things +so that we can be as transparent as possible. +We have a strong security & privacy focus for all our systems +so all private backend systems like databases are always locked down. + +As always, if you have a security question or concern, +please contact us responsibly. + diff --git a/postgres/backup-fly-postgres.md b/postgres/backup-fly-postgres.md index e1eea0c..ca59f33 100644 --- a/postgres/backup-fly-postgres.md +++ b/postgres/backup-fly-postgres.md @@ -1,9 +1,9 @@
-# How to Backup Fly.io Postgres Database +# How to Backup `Fly.io` `Postgres` Database A comprehensive step-by-step guide -to backing-up your Fly.io `Postgres` database +to backing-up your `Fly.io` `Postgres` database on your `localhost`.
@@ -12,11 +12,22 @@ on your `localhost`. You need to get the data from a Fly.io `Postgres` instance. +_Your_ reason may be different, +see our [context](#context-) below. + +## What? 🤔 + +Backup your `Postgres` DB running on `Fly.io` +and use the data somewhere `else`; +in our case we are migrating our DBs to `Hetzner` +where we have a +[high availability](https://en.wikipedia.org/wiki/High_availability) +cluster. ## How? 👩‍💻 ### 0. Before You Start -n + Before you attempt to access the `Postgres` database on `Fly.io`, ensure you are authenticated with your `Fly.io` account; run the command: @@ -79,29 +90,39 @@ pg_dump -h localhost -U hits_e2k5m6j4k46d0v7p -d hits --verbose > backup.sql > **Note**: If you need to get the password for the `Postgres` instance, use the following command: + ```sh flyctl ssh console -a hits -C "printenv DATABASE_URL" ``` + > in our case it was: + ```sh flyctl ssh console -a hits -C "printenv DATABASE_URL" ``` + We saw: + ```sh postgres://hits_e2k5m6j4k46d0v7p:baf3d9f0bdf155bfetc@hits-db.internal:5432/hits?sslmode=disable ``` + Where the first section `postgres://` is the protocol, the `hits_e2k5m6j4k46d0v7p` is the DB username, `baf3d9f0bdf155bfetc` is the password and `hits` is the name of the database. Ref: https://community.fly.io/t/how-to-view-environment-variables-in-a-fly-machine/10830/2 + Once you have the password, export it as an environtment variable: + ```sh export PGPASSWORD="$put_here_the_password" ``` -> in our case it was: + +in our case it was: + ```sh export PGPASSWORD="baf3d9f0bdf155bfetc" ``` @@ -114,7 +135,7 @@ Once the `pg_dump` command finishes, proceed to the next step. ### 3. Close your port forwarding -Kill the connection to the `Fly.io` instance +Kill the connection to the `Fly.io` instance using keyboard shortcut: `Ctrl` + `C` (twice). ### 4. Restore your local database @@ -123,7 +144,6 @@ To restore the database you just backed up to `Postgres` running on your `localhost`, you _first_ need to ensure that `Postgres` is indeed running! - With the `backup.sql` on your `localhost`, run the following command in the working directory: @@ -158,7 +178,7 @@ e.g: http://localhost:8081/# image -i.e. there have been 3 page views on https://github.com/dwyl/start-here since we did the SQL dump a few mins ago. +i.e. there have been 3 page views on https://github.com/dwyl/start-here since we did the SQL dump a few mins ago. We can check the "live" count at: https://hits.dwyl.com/dwyl/start-here.svg e.g: ![hits-start-here-svg](https://hits.dwyl.com/dwyl/start-here.svg) diff --git a/postgres/migrate-db.md b/postgres/migrate-db.md new file mode 100644 index 0000000..5301991 --- /dev/null +++ b/postgres/migrate-db.md @@ -0,0 +1,56 @@ +
+ +# Migrate `Postgres` DB to `Hetzner` Cluster + +
+ +Migrate/restore a snapshot of `Postgres` Database +from `Fly.io` (unreliable) to a +[high availability](https://en.wikipedia.org/wiki/High_availability) +cluster +running on `Hetzner`. + +## 0. Before You Start: Get the Snapshot + +We wrote _detailed_ instructions for backing up +a `Postgres` DB running on `Fly.io`, +see: +[postgres/backup-fly-postgres.md] + +backup.sql + +With the `backup.sql` on your `localhost`, +you can start. + +## 1. Connect To `Hezner` VPS Using `Cyberduck` + +There are several ways to upload large files to a remote server, +we've been using +[`Cyberduck`](https://en.wikipedia.org/wiki/Cyberduck) +for the past few decades and it works very well. +It uses +[`SFTP`](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol) +to securely transfer files +and is Open Source: +[github.com/iterate-ch/cyberduck](https://github.com/iterate-ch/cyberduck) + +> The Official Docs are great: +[docs.cyberduck.io](https://docs.cyberduck.io/cyberduck/) +and if you get stuck, +just Google: +[google.com/search?q=cyberduck+tutorial](https://www.google.com/search?q=cyberduck+tutorial) + +Open `Cyberduck` +and navigate to the `/tmp` directory of the VPS: + +hits-upload-backup-to-hetzner + +Drag the `backup.sql` file from the `finder` window on `localhost` +to the `Cyberduc` window to start the upload. + +hits-upload-backup-in-progress + +Take a screen break and refill your water bottle +while you wait for upload to complete. + +hits-upload-complete \ No newline at end of file