During the Covid-19 pandemic I decided to set up a server at home for different use cases. The server, that I very creatively called server1, should be a central place to build a file archive, a git server (an alternative for GitHub projects not intended for the web), a testing server for new applications (such as Keestash) and other stuff. Further, server1 ist only intended to stay in the local intranet (not reachable from the web) and does not underly strict security constraints. Let’s talk about server1, the setup and especially configuration.
The hardware
As I said, one of the primary use cases for server1 is a web based file archive. Therefore, it is necessary to have a powerful computer with an external hard drives as a RAID.
The Minis Forum PC is shipped with an Intel Quad Core CPU, 4 GB RAM and 64 internal memory and pre installed Windows 10. The ICY Box case holds two 4GB Seagate HDD’s and serves as a RAID. You can find them on Amazon here, here and here.
ICY Box RAID Configuration
For those who do not know what a RAID is, Wikipedia describes it as:
RAID (“Redundant Array of Inexpensive Disks” or “Redundant Array of Independent Disks“) is a data storage virtualization technology that combines multiple physical disk drive components into one or more logical units for the purposes of data redundancy, performance improvement, or both.
Wikipedia description for RAID
Since I want to use the hard drive of the ICY Box mainly for archiving files and pictures, it is definitely useful to utilize a RAID for backup and data redundancy. There are a few hardware buttons to click for enabling the RAID mode. Please check the manual shipped with the Box and do not forget to format the hard drives with the appropriate file system (ext4 in my case).
Minis Forum PC Configuration
As I want to run a linux server on the PC, it is first necessary to format the hard drive and install Linux. I decided to use Debian as the operating system since it is the mostly used OS for servers and has a lot of packages/software to use.
To install Debian, we first need to create an bootable medium (e.g. USB memory stick). Debian images are available on the official Debian website’s download section and using UNetbootin, we can easily create a bootable medium to install Debian.
The Debian installation is very straightforward, except the network drivers for WiFi. Since they are not installed with the operating system, it is necessary to install them manually. There are mainly two steps to take into account, to install them. I described them here under “Linux Related Issues” and also uploaded the necessary text file here.
Debian Software and Service Management
So far, we enabled the RAID mode, formatted the hard drives in our ICY Box and installed Debian on the Minis Forum PC. The PC is connected to our local network (LAN or WiFi) and we have SSH access to the server from our laptop.
At this point, we could start to install all necessary software and services manually. For my case, I need the following software/services on my server:
- Apache: a webserver to host and test “internal” applications
- GIT: for software versioning, needed for Gogs
- PHP: to testing applications on “real” Linux hardware
- ownCloud: to have a web based file storage
- MySQL: a DBMS for the applications mentioned above
- Gogs: GitHub like git service to host private repositories
Further, some of the software packages mentioned above need extensions (such as curl for PHP), the ICY Box has to be mounted properly, the operating system users are available and have sufficient permissions and all files and folders, required to run software/services mentioned above are there and have the appropriate permissions.
It is obvious that configuration becomes much more complex and time consuming as the list of required software and services grows. Further, we need fallbacks and make a lot of security checks (for instance, checking file/folder and permission presence) in order to have a running system. It is clearly obvious that we need a way to automate these steps.
Puppet Apply
Puppet is a straightforward way to automate server configuration for installing and managing software as described above. There are two main modes in which you can run puppet: Master-Agent and Apply. In Master-Agent mode, the master holds and compiles the “puppet receipt”. Each agent requests periodically the receipt from the master and applies it, if there is any difference to the current state. The apply mode is a single server infrastructure, meaning that the server holds and compiles the receipt for itself. There is no outbound reporting to other machines by default. Puppet Apply acts like a standalone combination of Master-Agent.
Since I run a single server only, I am using Apply for my home server.
In Puppet context, we are talking about manifests when we want to configure our server. Configuration steps are defined in one or more manifest files which are then compiled to a single manifest and applied by puppet.
The use case I have – installing and managing software, services, users and files/directories – enables me to create a single manifest file. When the manifest is growing in future, it is possible to split it into logically separated manifests.
Further, if one or a collection of manifests solve a specific problem, it is possible to turn them into a puppet module and make it available for others.
Talking About Puppet Modules
Common problems are usually solved by others and are usuable by puppet as a module. For my use case, there are modules for apache, git, php and mysql. For instance, if you want to install the puppetlabs/apache module, you can simply run the following command: puppet module install puppetlabs-apache –version x.y.z
Puppet Receipt
Let’s talk about my puppet receipt. As mentioned above, I decided to create a single receipt file, mostly because there are common packages to install and I used it for a “learning by doing” process. Therefore, when talking about the receipt and commands, I am referring to a single file site.pp.
I strongly believe in object oriented programming and value advantages of classes and objects. I am aware of that server configuration with puppet is not comparable with OO, but still: I decided to utilise classes and want to see what advantages and possibilities they provide. Therefore, my whole configuration remains in a single init class. Depending on how configurable Puppet is in its nature, I want later split everything logical structured classes (once the receipt starts growing). But let’s have a look into the init class in site.pp:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 | class init { # play around package { 'curl': ensure => 'installed', } package { 'openssh-server': ensure => 'installed', } package { 'ca-certificates': ensure => 'installed', } # installing apache class { 'apache': default_vhost => true, mpm_module => false, # still dont know what it does but is needed to install itk module which is necessary for php } apache::vhost { 'owncloud': port => '80', docroot => '/var/www/html/owncloud', docroot_owner => 'www-data', docroot_group => 'www-data', servername => 'xxx.xxx.x.xxx/owncloud', } # order matters !! class { 'apache::mod::itk': } class { 'apache::mod::php': php_version => "7.1" } class { '::php::globals': php_version => '7.1', config_root => "/etc/php/7.1" } -> class { '::php': ensure => latest, manage_repos => true, fpm => false, dev => false, composer => true, pear => true, phpunit => false, apache_config => true, extensions => { zip => { ensure => latest, provider => apt, }, xml => { ensure => latest, provider => apt, }, mbstring => { ensure => latest, provider => apt, }, gd => { ensure => latest, provider => apt, }, curl => { ensure => latest, provider => apt, }, pdo-mysql => { ensure => latest, provider => apt, }, pdo => { ensure => latest, provider => apt, }, json => { ensure => latest, provider => apt, }, mysql => { ensure => latest, provider => apt, }, }, } class { '::mysql::client': package_name => 'mysql-client', bindings_enable => true, } class { '::mysql::server': package_name => 'mysql-server', root_password => 'top_secret', remove_default_accounts => true, service_name => 'mysql', create_root_my_cnf => true, create_root_user => true, manage_config_file => true, config_file => '/etc/my.cnf', purge_conf_dir => true, restart => true, override_options => { mysqld => { bind-address => '0.0.0.0', datadir => '/var/lib/mysql', log-error => '/var/log/mysqld.log', pid-file => '/var/run/mysqld/mysqld.pid', wait_timeout => '600', interactive_timeout => '600', server-id => '1', log-bin => 'mysql-bin', relay-log => 'mysql-relay-log', auto-increment-offset => '1', auto-increment-increment => '2', }, mysqld_safe => { log-error => '/var/log/mysqld.log', }, }, } -> mysql_user { "owncloud@localhost": ensure => present, password_hash => mysql_password("top_secret"), } -> mysql_user { "gogs@localhost": ensure => present, password_hash => mysql_password("top_secret"), } mysql::db { 'owncloud': user => 'owncloud', password => 'top_secret', host => 'localhost', grant => ['ALL'], charset => 'utf8', } mysql::db { 'gogs': user => 'gogs', password => 'top_secret', host => 'localhost', grant => ['ALL'], charset => 'utf8', } mount { '/media/external1': ensure => 'mounted', device => '/dev/sda1', fstype => 'ext4', options => 'rw', } file { '/media/external1/owncloud': ensure => 'directory', owner => 'www-data', mode => '0755', } file { '/media/external1/owncloud/data': ensure => 'directory', owner => 'www-data', mode => '0755', } file { '/var/www/html/owncloud': ensure => 'directory', owner => 'www-data', mode => '0770', } user { 'gogs_user': ensure => 'present', comment => 'global user for gogs system', managehome => true, password => 'top_secret', } } class { 'init': } |
Explanations
The first three package commands manage packages from the official OS repositories (apt in this case). We are ensuring that curl, openssh-server and ca-certificates are installed on the system.
Next, we are using a class called ‘apache’ and located in another module. Notice that we do “only” care about how apache is configured, the most work however is done under the hood by the module.
The apache::vhost resource is interesting: As I mentioned earlier, I want to run a web based file storage. Therefore, I need to configure apache vhost and define some values (the incoming port, document root, webserver user, etc).
Next, we are configuring PHP. We need to enable the PHP module for apache which is done on line 33. From line 41 to 89, we ensuring that PHP is installed in version 7.1 together with some extensions.
Lines 96-125 install and configure MySQL and 126-143 create and monitor database users and schemas together with credentials and permissions.
Line 151 is interesting as it ensures a mount: As described above, we are using a RAID for file storage. mount ensures that the RAID is mounted properly and the file commands in 158, 164 and 170 ensure that the corresponding directories exist.
Lastly, we create an Linux user which is needed by Gogs and call our init class on line 185.
Notice that modules for packages such as owncloud or gogs are not available/in a beta state and are therefore unfortunately istalled manually.
IMPORTANT: As mentioned in the beginning, this server remains in my local network and is never accessible from the internet. Therefore, I did not care about secure communication and encryption (such as SSL/TLS). Please keep this in your mind while configuring your server – especially if your server is intended for the public web.
Once the receipt is ready, the installation on the server is very straightforward. We need to store the file on the server and run the following command:
1 | /opt/puppetlabs/bin/puppet apply /path/to/site.pp |
Conclusion
In short: it work’s once the receipt is written properly. While dealing with puppet, I noticed that there is not as much experience as with other technologies (for example, typing “puppet” into StackOverflow has about 11,000 unanswered questions where as JavaScript results in 582,000 unanswered questions). Therefore, you have to make a deeper look into things when having a problem and want to solve it. Try to make notes as the solutions found will not be the first entry in the Google search and blog about it to make it available for others 🙂