Code Monkey home page Code Monkey logo

os_preparation's Introduction

Table of Contents

Try Ubuntu now

Try Ubuntu World ===> https://github.com/charlietag/ubuntu_preparation

CentOS Linux Server OS Preparation

You want initialize your linux server by your own script. But you DO NOT want to use PUPPET , CHEF , Ansible. You can just leverage my initialization project here.

This is a small light bash project. Suit small companies which have only few servers to maintain. GIVE IT A TRY!!

centos stream release 9 server environment settings

  • This is useful when

    • You have less than 5 CentOS-Stream-9 servers to maintain.
    • You are deploying monolithic architecture app.

  • This repo is TOTALLY transfer from passenger to puma for rails.

    • NGINX + PUMA + PHP-FPM + MariaDB + Rails + Laravel + Redmine

Environment

  • CentOS Stream release 9

    • os_preparation
      • release : master v3.x.x

  • CentOS Stream release 8

    • os_preparation
      • release : v2.x.x

  • CentOS 8 (8.x)

    • os_preparation
      • release : v1.x.x

  • CentOS 7 (7.x) (deprecated)

    • os_preparation
      • release : v0.x.x

  • CentOS 7 (7.x) - passenger + nginx version (deprecated)

    • os_preparation

      • release : before_passenger_to_puma

      • If you prefer passenger + nginx (passenger-install-nginx-module)

        please switch to git tag named "before_passenger_to_puma"

        git clone --depth 1 --branch before_passenger_to_puma https://github.com/charlietag/os_preparation.git

Notice

  • Due to CentOS 8 - EPEL-modular repo always failed everytime updating metadata cache. This will disable repo cache expiration permanently, instead, using dnf-automatic / yum-cron to makecache
    • [Optional] Make cache before doing DNF / YUM installation
      • /root/bin/dnf.sh (alias dnf) will help you with this
      • dnf makecache / yum makecache
    • Disabled
      • dnf repo cache expiration
        • dnf config-manager --setopt metadata_expire=-1 --save
    • Enabled
      • dnf-automatic
        • For dnf makecache by daily
  • Before os_security
  • Systemd target
    • Default target (os_preparation will force to use this target)
      • multi-user.target
    • Ref. /etc/inittab
      • multi-user.target: analogous to runlevel 3
      • graphical.target: analogous to runlevel 5
    • WARNING If you are under graphical.target NOT under multi-user.target.
      • It is highly recommended that you do the following:
        • Reinstall whole CentOS using "Minimal Install" / "Server"
      • Or at least try the following:
        • systemctl set-default multi-user
        • reboot
        • dnf groupinstall 'Minimal Install' (If this is not working, try also dnf groupinstall 'Server')
        • dnf groupremove 'Server with GUI'
        • reboot
        • you can start with os_prepation now
      • Reference description here
    • Check method
      • systemctl get-default
  • Environment Groups
    • Based on (os_preparation will make sure this environment group is installed)
      • "Minimal Install"
    • Removing Environment Groups (os_preparation will remove these environment groups by default)
      • "Server with GUI"
      • "Workstation"
      • "KDE Plasma Workspaces"
      • "Virtualization Host"
      • "Custom Operating System"
    • Check method
      • dnf grouplist

Warning

  • Please do this in fresh install OS
  • What does this not cover, DO the following manually
    • Login user
      • Change password of root
      • Add GENERAL USER and setup password of GENERAL USER
    • /etc/ssh/sshd_config
      • PermitRootLogin no
      • PasswordAuthentication yes
    • RAM

      • mkswap if RAM is insufficient to start MariaDB

        mkdir /swap
dd if=/dev/zero of=/swap/swapfile bs=1M count=4096
mkswap /swap/swapfile
chmod 0600 /swap/swapfile
/sbin/swapon /swap/swapfile
        chmod 755 /etc/rc.d/rc.local
echo "/sbin/swapon /swap/swapfile" >> /etc/rc.d/rc.local

Configuration

  • ssh without SendEnv

    • command ssh

      • ssh_to

      • config /etc/ssh/ssh_config

        # SendEnv LANG LC_*

    • iTerm2 setting iterm2_disable_setting_LC_ALL.png

  • Before installation

    dnf clean all
dnf install -y git
git clone https://github.com/charlietag/os_preparation.git

  • Make sure config files exists , you can copy from sample to modify.

    cd databag
ls |xargs -I{} bash -c "cp {} \$(echo {}|sed 's/\.sample//g')"

  • Mostly used configuration :

    • DEV use (server in Local / server in Cloud) && Production use (server in Local / server in Cloud)

      databag/
├── F_01_ENV_02_os_01_env.cfg
├── F_01_ENV_04_ssh_config.cfg
└── _gitconfig.cfg

  • Verify config files (with syntax color).

    cd databag

echo ; \
ls *.cfg | xargs -I{} bash -c " \
echo -e '\e[0;33m'; \
echo ---------------------------; \
echo {}; \
echo ---------------------------; \
echo -n -e '\033[00m' ; \
echo -n -e '\e[0;32m'; \
cat {} | grep -vE '^\s*#' |sed '/^\s*$/d'; \
echo -e '\033[00m' ; \
echo "

  • Verify ONLY modified config files (with syntax color).

    cd databag

echo ; \
ls *.cfg | xargs -I{} bash -c " \
echo -e '\e[0;33m'; \
echo ---------------------------; \
echo {}; \
echo ---------------------------; \
echo -n -e '\033[00m' ; \
echo -n -e '\e[0;32m'; \
cat {} | grep -v 'plugin_load_databag.sh' | grep -vE '^\s*#' |sed '/^\s*$/d'; \
echo -e '\033[00m' ; \
echo "

Easy Installation

I'm a lazy person. I want to install ALL and give me default configurations running Nginx , MariaDB, php-fpm, puma 5 (rails). And help me to create default projects about "Rails" and "Laravel"

  • Command

    ./start -a
reboot

  • Default project path

    • DEFAULT user for rails/laravel developer is not ssh allowed

      • /etc/ssh/sshd

        DenyGroups no-ssh-group

    • group "no-ssh-group" add to default dev user

      • phpuser (this name can be modified)
      • rubyuser (this name can be modified)

    • rails

      • default user: rubyuser (can be changed)
    /home/${current_user}/rails_sites/myrails/
--->
/home/rubyuser/rails_sites/myrails/
    • Redmine
      • default user: rubyuser (can be changed)
    /home/${current_user}/rails_sites/redmine/
--->
/home/rubyuser/rails_sites/redmine/
    • laravel
      • default user: phpuser (can be changed)
    /home/${current_user}/laravel_sites/myrails/
--->
/home/phpuser/laravel_sites/myrails/

  • Config your own hosts file (/etc/hosts)

    <192.168.x.x> myrails.centos8.localdomain
<192.168.x.x> redmine.centos8.localdomain
<192.168.x.x> mylaravel.centos8.localdomain

  • Browse URL

    http://myrails.centos8.localdomain
http://redmine.centos8.localdomain (default account: admin/admin)
http://mylaravel.centos8.localdomain

Advanced Installation

I want to choose specific part to install.

  • Command

    ./start.sh -h
usage: start.sh
  -a                   ,  run all functions
  -i func1 func2 func3 ,  run specified functions

Customize your own function

Folder

  • functions/

    • Write your own script here, file named start with F_[0-9][0-9]_YourOwnFuntionName.sh

    • Run command

      ./start.sh -i YourOwnFuntionName

  • templates/

    • Put your own templates here, folder named the same as YourOwnFuntionName

  • databag/

    • Put your special config variables here, file named the same as YourOwnFuntionName
    • How to use

      • In databag/YourOwnFunctionName

        • local your_vars_here

      • In templates/YourOwnFunctionName/yourowntemplate_file

        • You can use ${your_vars_here}

      • In YourOwnFuntionName , you can call

        # Method : eval "echo \"$variable\""
# Might have escape issue, if template is complicated
RENDER_CP ${$CONFIG_FOLDER}/yourowntemplate_file /SomeWhere/somewhere
        # Method : cat template | sed 's/\{\{var\}\}/$var/g'
# BETTER method for rendering template
RENDER_CP_SED ${$CONFIG_FOLDER}/yourowntemplate_file /SomeWhere/somewhere

        instead of

        cp ${$CONFIG_FOLDER}/yourowntemplate_file /SomeWhere/somewhere

      • In YourOwnFuntionName , you just want to LOAD VARIABLES ONLY from databag, try add a comment into your function script

        # For Load Variables Only Usage, add the following single comment line with keyword DATABAG_CFG:enable
# DATABAG_CFG:enable

  • helpers/

    • Write your own script here, file named start with helper_YourOwnHelperName.sh
    • Works with helpers_views

  • helpers_views/

    • Put your own templates for ONLY helper USE here, folder named the same as YourOwnHelperName

  • tasks/

    • Write your own script here, file named start with task_YourOwnTaskName.sh , _task_YourOwnTaskName.sh
    • Scripts here will automatically transfer to function, just like scripts under "functions/"
    • But this is for global use for os_preparation , os_security. So it's been moved to os_preparation_lib

  • plugins/

    • Only scripts which can be called everywhere like, ${HELPERS}/plugins_scripts.sh
    • Use this as a script, not function

Predefined variables

(root)# ./start.sh -i F_00_debug
#############################################
         Preparing required lib
#############################################
Updating required lib to lastest version...
Already up to date.

#############################################
            Running start.sh
#############################################

---------------------------------------------------
NTP(chrony) ---> pool.ntp.org
---------------------------------------------------
RUN: chronyd -q 'pool pool.ntp.org iburst'
2020-09-08T01:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASH +IPV6 +DEBUG)
2020-09-08T01:47:38Z System clock wrong by -0.002320 seconds (step)
2020-09-08T01:47:38Z chronyd exiting
RUN: hwclock -w
---------------------------------------------------


==========================================================================================
        F_00_debug
==========================================================================================
-----------lib use only--------
CURRENT_SCRIPT : /root/os_preparation/start.sh
CURRENT_FOLDER : /root/os_preparation
FUNCTIONS      : /root/os_preparation/functions
LIB            : /root/os_preparation/../os_preparation_lib/lib
TEMPLATES      : /root/os_preparation/templates
TASKS          : /root/os_preparation/../os_preparation_lib/tasks
HELPERS        : /root/os_preparation/helpers
HELPERS_VIEWS  : /root/os_preparation/helpers_views

-----------lib use only - predefined vars--------
FIRST_ARGV     : -i
ALL_ARGVS      : F_00_debug

-----------function use only--------
PLUGINS            : /root/os_preparation/plugins
TMP                : /root/os_preparation/tmp
CONFIG_FOLDER      : /root/os_preparation/templates/F_00_debug
DATABAG            : /root/os_preparation/databag
DATABAG_FILE       : /root/os_preparation/databag/F_00_debug.cfg

-----------function extended use only--------
IF_IS_SOURCED_SCRIPT  : True: use 'return 0' to skip script
IF_IS_FUNCTION        : True: use 'return 0' to skip script
IF_IS_SOURCED_OR_FUNCTION  : True: use 'return 0' to skip script

${BASH_SOURCE[0]}    : /root/os_preparation/functions/F_00_debug.sh
${0}                 : ./start.sh
${FUNCNAME[@]}          : source F_00_debug L_RUN L_RUN_SPECIFIED_FUNC source source main
Skip script sample    : [[ -n "$(eval "${IF_IS_SOURCED_OR_FUNCTION}")" ]] && return 0 || exit 0
Skip script sample short : eval "${SKIP_SCRIPT}"

================= Testing ===============
----------Helper Debug Use-------->>>

-------------------------------------------------------------------
        helper_debug
-------------------------------------------------------------------
HELPER_VIEW_FOLDER : /root/os_preparation/helpers_views/helper_debug


----------Task Debug Use-------->>>

-----------------------------------------------
        task_debug
-----------------------------------------------

Note

Installed Packages

Folder privilege

After this installation repo, the server will setup with "Nginx + Puma (socket)" , "Nginx + PHP-FPM (socket)" , so your Rails, Laravel, can run on the same server. The following is something you have to keep an eye on it.

  1. folder privilege

  • Rails Project

    rails new <rails_project> -d mysql --skip-spring
cd <rails_project>
chown -R ${current_user}.${current_user} log tmp

  • Laravel Project

    composer create-project --prefer-dist laravel/laravel <laravel_project>
cd <laravel_project>
chown -R ${current_user}.${current_user} storage
chown -R ${current_user}.${current_user} bootstrap/cache
  1. Command

  • Rails

    rails new <rails_project> -d mysql --skip-spring

  • Rails 5.1 has dropped dependency on jQuery, you might want it back via yarn

    1. Add npm of jquery using Yarn

      cd <rails_project>
yarn add jquery

    2. Setup jquery npm for asset pipeline

      vi <rails_project>/app/assets/javascripts/application.js
      //= require rails-ujs
//= require turbolinks
//= require jquery/dist/jquery
//= require bootstrap/dist/js/bootstrap
//= require_tree .

    3. Yarn works with rails 5.1 asset pipeline as below

    • Usage for default path: <rails_project>/node_modules/{pkg_name}/dist/{pkgname}.{js,css}

      //= require jquery

    • If package is different from this rule, ex: bootstrap. You might specify explicitly (better)

      //= require jquery
      //= require jquery/dist/jquery
//= require bootstrap/dist/js/bootstrap

  • Laravel

    composer create-project --prefer-dist laravel/laravel <laravel_project>

  • Useful script snippet

    • If you are always get disconnected, and you want to kill last failed connection of SSH

      netstat -palunt |grep -i est | awk '{print $7}'| cut -d'/' -f1 |xargs -I{} bash -c "ps aux |grep sshd |grep {}|grep -v grep" | head -n -1 | awk '{print $2}' |xargs -I{} kill {}

    • If you want to restart network for new config, instead of using systemctl restart network, which is deprecated in CentOS 8

      • Reload network config (mostly, this would work)

        nmcli c reload

      • Stop networking and start networking in NetworkManager (NM)

        nmcli n off; nmcli n on

    • Since RHEL 9 , no more configs under /etc/sysconfig/network-scripts/, instead, keyfile under /etc/NetworkManager/system-connections only. So config network using nmcli will be a better method

    • Modify network static IP using nmcli

      • Setup static ip

        nmcli connection modify eth0 \
  ipv4.addresses 192.168.122.7/24 \
  ipv4.gateway 192.168.122.1 \
  ipv4.dns 192.168.122.1 \
  ipv4.method manual

      • Disable IPv6, and peerDNS

        nmcli connection modify eth0 \
  ipv4.ignore-auto-dns "true"

nmcli connection modify eth0 \
  ipv6.method "disabled" \
  ipv6.addr-gen-mode "stable-privacy" \
  ipv6.ignore-auto-dns "true" \
  ipv6.ignore-auto-routes "true" \
  ipv6.never-default "true"

      • List only device name except loop 0 using nmcli

        nmcli -g name connection show

Ruby gem config

  • gem install without making document

    • Deprecated

      no-ri, no-rdoc

    • Config

      echo "gem: --no-document" > ~/.gemrc

Database configuration for production

  • Remove test database and setup root password

    After doing this, still need some tweak, try to manage database with https://www.adminer.org/

    $ mysql_secure_installation

    Just keep hitting <ENTER>, to USE ALL DEFAULT SETTING

  • After mysql_secure_installation

    • MariaDB 10.5 auth method will just like MariaDB 10.3

  • Database tools - Adminer

Extra functions

  • RENDER_CP

    • Render template using eval (Might have escape issue, if template is complicated)

      # Method : eval "echo \"$variable\""

    • Sample

      • databag
      local var="Hello World"
      • template (${$CONFIG_FOLDER}/yourowntemplate_file)
      This is $var
      • function
      RENDER_CP ${$CONFIG_FOLDER}/yourowntemplate_file /SomeWhere/somewhere
      • result (/SomeWhere/somewhere)
      This is Hello World

  • RENDER_CP_SED

    • Render template using sed (BETTER method for rendering template)

      # Method : cat template | sed 's/\{\{var\}\}/$var/g'

    • Sample

      • databag
      local var="Hello World"
      • template (${$CONFIG_FOLDER}/yourowntemplate_file)
      This is {{var}}
      • function
      RENDER_CP_SED ${$CONFIG_FOLDER}/yourowntemplate_file /SomeWhere/somewhere
      • result (/SomeWhere/somewhere)
      This is Hello World

  • SAFE_DELETE

    • Check file names and path before rm any dangerous files, preventing from destoying whole server

      • check for the following dangerous key words

        .
..
*
/
.*
*.*
        "$(echo "$(find / -maxdepth 1 ;  readlink -m /* )" | sort -n | uniq)"

    • Sample

      # --- Should be failed ---
DELETE_FILE="/root/delete_me/.*"
# --- safe delete command usage ---
SAFE_DELETE "${DELETE_FILE}"

(Git) Stash details

  • Ref. https://git-scm.com/docs/git-stash

  • (Git) stash list

    $ git stash list
stash@{0}: WIP on redmine_4.0.7: a853fc0 Fix sort projects table by custom field (#32769).
stash@{1}: WIP on redmine_4.0.6: 22ebc68 tagged version 4.0.6
    • redmine_4.0.6 / redmine_4.0.7, these mean branch name
    • if you want to restore data, you'd better checkout the the related branch

  • Display all stash contents

    git stash list | cut -d':' -f1 | xargs -I{} bash -c "\
  echo; \
  echo ----------------------------------------------- {} -----------------------------------------------;\
  git stash show -p {}; echo\
"

(Git) Push and Pull

  • Push git commits to remote

    git push

  • Push git tags to remote

    git push --tags

  • Fetch git commits to local

    git fetch

  • Fetch git tags to local

    git fetch --tags

  • Fetch git commits to local and then MERGE to Working Directory

    git pull

Upgrading Redmine

Reference

Backup current redmine

  • Database
    • mysqldump -u {db_user} -p --lock-all-tables --skip-tz-utc redmine > redmine_$(date +"%Y%m%d")_skip-tz-utc.sql
  • Application & files
    • cp -a redmine redmine_bak

Customized files

  • plugins
    • /home/rubyuser/rails_sites/redmine/plugins/redmine_*
  • themes
    • /home/rubyuser/rails_sites/redmine/public/themes/{a1,circle,PurpleMine2}
  • session token
    • /home/rubyuser/rails_sites/redmine/config/initializers/secret_token.rb
  • uploaded files
    • /home/rubyuser/rails_sites/redmine/files/

(Method 1) Upgrading from a git checkout

  • Stop puma server

    • puma-systemd-mgr -p -i redmine

  • Go to the Redmine root directory and run the following command:

    cd redmine
git stash
    git checkout master
git fetch
git fetch --tags
git pull

    sometimes git pull will not fetch tags, instead, we need to fetch tags by git fetch --tags

    especially when tags name or tags <-> commit , has been changed

    git co 4.0.7 -b redmine_4.0.7
git stash pop
git status |grep 'both modified:' |awk '{print $3}' |xargs -I{} bash -c "echo --- git reset HEAD {} ---; git reset HEAD {}"

  • Fix conflicts

  • Perform the upgrade

    # gemset name using redmine version
echo "gemset_redmine_4.1.0" > .ruby-gemset

# switch to the new gemset
cd
cd -

# Update gem / bundler for this gemset
gem update --system
gem install bundler

# Install the required gems by running the following command
bundle update

# Update the database
bundle exec rake db:migrate RAILS_ENV=production
bundle exec rake redmine:plugins RAILS_ENV=production

# Clean up
bundle exec rake tmp:cache:clear RAILS_ENV=production

  • Start puma server

    • puma-systemd-mgr -s -i redmine

  • Go to "Admin -> Roles & permissions" to check/set permissions for the new features, if any.

  • Finally, clear browser's cached data (To avoid strange CSS error)

    • Chrome -> History -> Clear History -> Choose ONLY "Cached images and files"

(Method 2) Upgrading from a fresh installation

  • Stop puma server

    • puma-systemd-mgr -p -i redmine

  • Backup current redmine

  • Remove the following lines from script functions/F_02_PKG_06_ruby_09_redmine_create.sh (F_02_PKG_06_ruby_09_redmine_create_diff.png)

    if [[ -z "${redmine_db_pass}" ]]; then
  mysql -u root -e "CREATE DATABASE ${redmine_db_name} CHARACTER SET utf8;"
else
  mysql -u root -p${redmine_db_pass} -e "CREATE DATABASE ${redmine_db_name} CHARACTER SET utf8;"
fi
    su -l $current_user -c "cd ${redmine_web_root} && bundle _${this_redmine_bundler_version}_ exec rake generate_secret_token"
    if [[ -n "${redmine_default_lang}" ]]; then
  su -l $current_user -c "cd ${redmine_web_root} && bundle _${this_redmine_bundler_version}_ exec rake redmine:load_default_data RAILS_ENV=production REDMINE_LANG=${redmine_default_lang}"
fi

  • Perform the fresh installation

    • ./start -i F_02_PKG_06_ruby_09_redmine_create

  • Restore files from backup

    • redmine/config/initializers/secret_token.rb
    • redmine/files/

  • Start puma server

    • puma-systemd-mgr -s -i redmine

  • Go to "Admin -> Roles & permissions" to check/set permissions for the new features, if any.

  • Finally, clear browser's cached data (To avoid strange CSS error)

    • Chrome -> History -> Clear History -> Choose ONLY "Cached images and files"

Upgrading MariaDB

For some cases, we need to upgrade MariaDB without data lost. Here is my note about this.

Reference (mariadb.com)

How to Upgrade

  • Backup current database

    # mysqldump -u root -p --lock-all-tables --skip-tz-utc -A > all_`date +"%Y%m%d"`_skip-tz-utc.sql

  • Stop MariaDB

    # systemctl stop mariadb

  • Uninstall the old version of MariaDB

    dnf remove -y MariaDB-common MariaDB-client MariaDB-shared MariaDB-server MariaDB-devel

  • Modify the repository configuration to newer version

  • Install the new version of MariaDB

    dnf install -y MariaDB-common MariaDB-client MariaDB-shared MariaDB-server MariaDB-devel

  • Make any desired changes to configuration options in option files, such as my.cnf. This includes removing any options that are no longer supported.

    # cat /etc/my.cnf.d/server.cnf | grep -B1 '127.0.0'
[mysqld]
bind-address = 127.0.0.1

  • Start MariaDB

    # systemctl start mariadb

  • Run mysql_upgrade

    • mysql_upgrade does two things:
      • Ensures that the system tables in the#mysql database are fully compatible with the new version.
      • Does a very quick check of all tables and marks them as compatible with the new version of MariaDB .
    • mysql_upgrade -u root -p

      • After this command, there would be a file generated for letting you know this database has already been upgraded. (owner of the file is root)

        $ ls /var/lib/mysql | grep upgrade
-rw-r--r--  1 root  root    15 Sep  9 14:24 mysql_upgrade_info
        $ cat mysql_upgrade_info
10.5.5-MariaDB

  • Restart MariaDB - Done

    • It would be better to restart MariaDB, if it's allowed.

      # systemctl restart mariadb

CHANGELOG

os_preparation's People

Contributors

charlietag avatar devnodereact avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.