Wheel Meeting Minutes - Saturday 2020-04-18 14:00
=================================================

VENUE: https://meetings.ucc.asn.au/

*Meeting opened 14:06*

## Attendance
- Present
- \[TPG\]
- \[BRD\]
- \[BOB\]
- \[CFE\]
- \[333\]
- \[MPT\]
- \[MTL\]
- \[NTU\]
- \[SJH\]
- \[LE@\]

### Apologies
- None

### Absent
- None

## Schedule next meeting
- Schedule/delegate reminders of next meeting
- \[TPG\]: Monthly is working well
- \[BOB\]: Saturday May 23rd suggestion for next meeting
- Scheduled for 2020-05-24 14:00

## Standing items (brief)

### SWS
- Done

### Status Check: Regular Updates
- eg. Debian Oldstable 9 "Stretch" --> Debian Stable 10 "Buster"
- Discord-irc.ucc.asn.au could use a rebuild, it's Jessie
- OcsInventory, uccmonitor (see https://wiki.ucc.asn.au/MissionControl) for an overview
- Out-of-date servers:
     - meersau (stretch) 
     - discord-irc (jessie)
     - gitlab ()
     - unisfa-koha (stretch)
     - *Contact Blair, Chase, or Felix from UniSFA to arrange downtime*
     - salmon (? container, hard to upgrade)
     - mooneye (working on it) 
     - *Dead disk, needs replacing or new machine STAT*
     - mussel (stretch)
     - murasoi (stretch)
     - motsugo (stretch)
- Tuesday 21 Apr upgrade event - "Let's Break UCC"

### Status Check: Password/Key Rotations
- DEFER
- https://en.wikipedia.org/wiki/Pro_re_nata

### Status Check: Backups
- \[NTU\]: We should have done that offsite file-restore demo
- Communal decision to remove off mollitz webcam-archives
- Full disks
     - \[TEC\]: If we can clean the old warnings, include webcamrestore, we can get timely new uccmonitor alerts
- Dead disks
- Mollitz (Dell 2950) can't have disks larger than 2TiB, and is old, so we should look at replacing 
- Other options:
     - Motsugo could be replaced
     - Old Motsugo coule become the new backup server
     - Worst case: could deploy the Cisco UCS220 or Mudkip
     - Mudkip has plenty of space drive caddies
     - Cisco for user server?
- ACTION: Get Cisco server to a workable state

## New Matters
- \[BRD\] allocating more memory to minecraft2019
- \[RME\] & \[333\]: Migrate to magikarp?
     - \[NTU\]: Save to molmol NFS or Ceph? (fast to migrate); or
     - Save to magikarp local Optane dm-cache? (fast to access)
- \[333\]: minecraft2019:/, currently hosted on loveday, is 128GB? Could be shrunk? Will take a while to migrate 
     - Run MineOS and a bunch of minecraft servers: maybe 128GB is right-sized?
     - Check with \[BRD\] & \[MDD\] about the rootfs shrinking?
- ACTION: \[333\] to migrate and look at shrinking system disk if necessary
- \[333\] Discuss current security of remote management, and whether we need to lock down the 192.168.2.0 subnet further (ie. no access from motsugo)
     - eg. some management controllers are positively ancient and are unpatched
- See this article for an example of a nasty bug (hint: we have 3 HPs):
     - https://it.slashdot.org/story/19/07/08/007202/critical-bug-last-year-allowed-bypassing-authentication-on-hpe-ilo4-servers-with-29-a-characters
- Wheel-only jumpbox or VPN? Log traffic to/from that subnet?
- ipmiview tool is handy? (By SuperMicro)
- ssh in to some of them - which ones to be decided. Probably runs dropbear
- ACTION: \[MTL\] will spin up a jumpbnox VM for testing
     - ...but a physical host is better than a V<
- \[MTL\] mailauesi - imaps, pop3s, submission: currently wheel-only logins
- Live testing: imap works, smtp doesn't?
     - Submission replaces smtp
- Deprecate secure.ucc.asn.au for people's mail clients
- https:gitlab.ucc.asn.au/ucc-systems-ucc-ansible-soe
- \[MTL\] maculatus - new home VM for flame
- libc6:i386 installed, old flame MudOS driver runs
- listenes via firewall rules to telnet port 4242, and many other ports (like ssh.ucc.asn.au)
- possibly no one uses most of those anymore?
- webtty
- \[MTL\] mailfish - new home for mail services from mooneye and mailman 
- postfix off switch, similar to postgrey, stop accepting deliveries if AD or NFS is down
- Remaining stuff on mooneye  - DNS (rearchitect? \[MTL\], see below)
- \[MTL\] Folding@Home
     - Rolled out on Clubroom Desktops
     - https://stats.foldingathome.org/team/261452
     - Ranked 11,724 of 251,202
- Windows Clubroom machines?
- Try desktop: Christmas
- WSL+OpenSSHd -> running chocolately
- can rdesktop in via guacamole/maaxen?
- ACTION: \[MTL\] to do above. 
- \[TEC\]: GPU Server
     - Bob raised on Facebook group: https://www.facebook.com/universitycomputerclub/permalink/3767237643318123
     - \[DBA\] David Adams, Cursor XYZ
- \[NTU\] Proxmox Cluster Upgrades and Updates
- \[CFE\] We have a virtual cluster we can test upgrading Proxmox PVE v5 -> v6

```
root@magikarp:~# pveversion
pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-24-pve)
```

- \[MPT\] 4G backup uplink
- Set up: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/2020-April/005323.html
- Can be used for outgoing with source-based policy routing
- Has CGNAT IPv4 /64 address?
- Probably don't want to fail2ban on that if it all comes from the one IP address 
- Needs a VPN?
- Wireguard?
- ACTION: \[MSH\] proof-of-concept on murasoi?
- TINC?

- Cloudflare! All! The! Things! ?!?!?!?
- Still Priority #1
- \[MTL\] UCC webservers, etc (was: mussel)_
- More SSL expiries coming in June
- If only we can make a CNAME change to ucc.gu.uwa.edu.au?
- letsencrypt might see our secondaries?
- ACTION: \[NTU\]\[MTL\]\[333\] try it out and extend our expiries
- Set up offsite ucc.asn.au nameserver
- We've been using and developing iodine
- \[MPT\] has moved his domains to Cloudflare Free tier
    - Using dynamic DNS, via Cloudflare API: token based, not ddclient?

## Matters Arising Previously

#### ACTION: Annual Account Locking - if it has not happened by now, it's overdue
- Are the rejoining member/password reset/new member account procedures going OK?
     - Committee: Make it an event!
     - Account Locking Stream in Charity UnVigil 2020-05-13 through 2020-05-16

#### ACTION: \[333\] to document remote management options for our critical servers
- Initial document in place, further testing needs to be done
- More details to be added 
- See /home/wheel/docs/RemoteManagement.org

#### ACTION: \[333\] to sort out iDRAC for Mooneye as a priority

- Live demo for mudkip! HP iLO
     - ssh
     - vsp: Virtual serial port
     - systemctl enable getty@ttyS1
     - power reset
- Mail server from scratch on it, and a point a ucc subdomain at it, by the end of this weekend (2020-03-22)
- \[333\]: Played with EC2 instance, set up a DNS A record for it, and ran push.sh
- Backed up mooneye:/etc/bind/domains to https://gitlab.ucc.asn.au/ucc-systems/ucc-domains.git, cloned it onto cloud-mooneye
- Copied the named.config.local across
- Commented out mooneye-specific parts (like LetsEncrypt stuff and other referenced secrets only on mooneye)
- Tried running zonemake.py, only to find that it needed a package installed
- Once package was installed, it still wouldn't work due to symlinks to more stuff on mooneye (like /home/other/www/members.conf)
- \[333\] work more independently gitlab vs zonemake.py vs AD getent
- \[MTL\] \[NTU\] separate authoritative and recursive - DJB was right!
- \[NTU\] has been testing knot-DNS

#### ACTION: \[333\]\[THA\]\[TEC\] to buy 1TB SSDs for magikarp + mudkip
- Passed by committee on 2020-10-04.txt
- Austin Computers: ~$500 Each? MLC, not SLC/QLC https://.www.austin.net.au
- Samsung-mz-76q1t0-2-5-1000-gb-serial-ata-iii-v-nand-mlc.html
- *But the MLC is a lie!*

*Meeting closed 17:04*