UCC Wheel Meeting on 21st March 2020

Minutes - Saturday 2020-03-21 14:00

Meeting opened 14:10

Attendance

  • Present
    • Mark Tearle [MTL]
    • Nick Bannon [NTU]
    • Dylan Hicks [333]
    • John Hodge [TPG]
    • Alex Thatcher [AJT]
    • Andrew Adamson [BOB]
    • Alfred Burgess [TBB]
    • Timothy E Chapman [TEC] (late at 3:09pm [less than an hour late])
    • James Andrewartha [TRS] (late at 3:44pm)
  • Apologies
    • Donald Sutherland [DAS]
    • James Arcus [MPT]
  • Absent

Schedule next meeting

  • Access to clubroom will limit physical meeting opportunities
  • Will brain storm other meeting and club opportunities with committee
  • Saturday 2020-04-18 14:00
    • the week after Easter

New Matters

Hardware Maintenance Issues

  • Fans for Magikarp. 12 fans are on order from Ebay via [333]
    • Concerns discussed about noise - no problem, clubroom empty
    • Worried about further failures

Clubroom Allocation

  • [MTL] Items from my personal notes

    • Use of ucc.asn.au domain name
    • Decentralising hosting / hybrid cloud
      • IP Space allocated to UCC
      • VPN solutions
        • IPSec (murasoi, broken)
        • OpenVPN (murasoi, needs local accounts)
        • ssh tunnels
        • PPTP (retired, insecure)
        • ACTION: This item is a PRIORITY. Schedule a working group to meet again to make progress here.

  • Discussion about service availability

    • Concerns about loveday, evil, merlo
    • Boot speed of mooneye
    • UCC AD on VM
    • Offline/disconnected copies of uccpass via git/rsync

  • Context of planning in response

    • Reprioritise in light of COVID19
    • Ensure UCC is well placed when we come back out of the next 6 months
    • Limited availability of volunteer time

  • Remote graphical access to UCC machines

    • [BOB] mentions Guacamole

COVID19

  • [MTL] Items from my personal notes
    • Backups
      • Current backup is to a machine at Christchurch Grammar School
      • Potential space for another machine there?
      • Around 12TB of data
      • Build up capability with VPN/storage to another location
        • AWS machine with minor services, except storage
        • Pricing of options
        • Basic services
          • Auth
          • Mail (Priority)
          • Shell
          • DNS (Priority)
          • VPN
          • Some routing?
        • Gratis first year of AWS service may be good for this
        • [MTL], [NTU] volunteers fixing UCC mail
        • What tells public DNS servers to do lookups from mooneye?
          • ucc.asn.au easy to repoint to new DNS server - log into our Namecheap DNS subscription
          • ucc.gu.uwa.edu.au - who do we talk to at Guild/UWA?
          • was Kelvin Lee at Guild, but the Guild is not making heavy use of the gu.uwa.edu.au or guild.uwa.edu.au subdomains
            • in practice, UWA IT
        • ACTION: [MTL], [NTU], and [TBB] to set up a regular chat to progress things
    • 3D Printers
      • Committee domain, punt to them
    • BigBlueButton
      • James has volunteered to set up a VM to do this
      • meetings.ucc.asn.au potential domain name?
      • Use for tech talks and online meetings
      • ACTION: [TRS] to set up
      • Maltair has tonnes of capacity to host this
      • LDAP auth to create rooms, anyone can join
    • Access to Clubroom desktop (was Folding@Home)
      • Could get idle clubroom machines simulating protein folding
      • Clubroom is closed, so could be a good use of compute
      • ACTION: [TEC] Change clubroom machines to boot into Linux as default
      • [333] Can remotely reboot Windows machines with shutdown -m <machine> -r -t <seconds>
    • UWA IT Contact
      • [TEC] to get in touch with IT, respond to Cloudflare email
      • [BOB] We need to organise a meeting with them to try and keep our autonomy
        • Generally, to make contact
        • Concretely, re: control over firewalling, mail filtering, etc.
      • Attendees:
        • [TEC]
        • [NTU]
        • and possibly [BOB] (with enough notice)
      • Questions to ask:
        • What threats they're targeting
        • How it works
        • How we can work with or around it
        • Cloudflare is actually issuing certs for *.ucc.gu.uwa.edu.au
          • We could block this with a CAA record
        • If compromised VM issue is raised: we have investigated and talked to the machine owner, we are about to delete that machine, do you have any objections or questions about it?
      • When?
        • Next week (starting Sunday 22/03/2020)
        • Later in day
      • ACTION: [TEC], [NTU], and [BOB] to email Matt DeGois and organise
    • Meetings, General Response (re. keeping club going)
      • See BigBlueButton
      • [TEC] and Jimbo have Cam. Hall access during the lockdown
    • Hardware status
      • See magikarp
      • Mooneye still needs replacing
        • TODO: Mirror DNS config to Murasoi
        • TODO: Find somewhere else to host mail, if it dies,
          • and forward port 25
          • Motsugo is a candidate
        • ACTION: [NTU] and [MTL]
    • Lights Out Management
      • ACTION: [333] to identify remote management options for our other servers
      • Add to existing wiki pages, or create new one for remote mgmt.
      • Mudkip and Magikarp are sorted with iLO
    • Access to Machine Room
      • [TEC] and [MPT] have been granted ?limited access to CH over the shutdown.

Services to other clubs

  • [MTL] Incoming emails
    • email for UniSFA Koha
    • ACTION: [MTL] to investigate

UWA DNS Poisoning of mooneye.ucc.gu.uwa.edu.au

  • https://crt.sh/?q=ucc.gu.uwa.edu.au
  • See discussion above

Standing items (brief)

New wheel members, additions, nominations

- Wheel Membership
    - Recruit new wheel members from UCC member base
    - Dadams [DBA] has expressed interest in joining
        - Points against: No free time
        - Technical competence: not bad
        - Interested in machine learning
        - ACTION: Ask him for a project that he'd like to do
        - [NTU] He was already asking about desktop GPU access: immediately add to winadmin & sprocket, ask him to setup WSL-opensshd on some desktops?
        - Re-evaluate wheel candidacy at next wheel meeting

Action items from previous meeting

  • Completed
    • [MTL] Start a tech@ discussion - done!
  • Outstanding and skipped due to time:
    • ACTION: [TEC] and [NTU] will automate email reminders
    • ACTION: [TEC] will probably send email reminders every Monday during March until it?s automated
    • ACTION: Add a README.home-and-away or similar to both skeleton directories
    • ACTION: LDAP/AD on uccmonitor
    • ACTION: minimal logging client and server setup - ansible: setup a new machine for central logging, journald/syslog forwarding
    • ACTION: Check our Letsencrypt Certificate Transparency logs: Which ones do we need to automate?
    • mooneye upgrade
    • ACTION: [MTL] Drive an email discussion: re: DNS resolver split for authoritative
    • ACTION: [MPT] Parts buys: Make shopping list: network, rails, storage - on tech@ list
    • ACTION: [LE@] Examine disc image on Monday and get back to UWA
    • Previous ACTION [???]: Figure out how to filter out configuration secrets from an etckeeper push mirror, a la http://svn.ucc.asn.au:8080/rancid/ucc/configs/
    • Previous ACTION: [???] Purchase a windows server licence - unsure if this can be done individually on connectingup.org

General discussion

  • ACTION: Annual account locking

  • [TEC][NTU] New service, uccmonitor (prometheus/grafana)

    • needs auth? and general adoption
    • worked example: add temperature monitoring to something?
      • disk latency, display in dashboard?
      • outbound SSH login attempts? TCP SYN rate?
      • inbound SSH login attempts?
    • free disc space - done?
    • [MTL] DNS queries/second
    • [DAA] active and available DHCP leases on the loft/wireless/clubroom networks

  • [LE@] VM needed patching --2020-02-10

    • double firewalled by UCC and UWA and shutdown
      • resolve incident with UWA, check firewall status
    • incident checklist/run-sheet
    • see guidelines below for proactive measures
    • [BOB] add VM to proxmox'ed marked-for-deletion pool- done!

  • [MTL][NTU] mooneye upgrades

    • DNS service before/during/after upgrade
    • Temporarily use winbind? avoid sssd < version 1.15.3: getent enumeration bug
    • extract flame to a VM?
    • extract wiki
      • to a VM?
      • or to molmol, close to where /services/wiki is stored and fast to bring up in outages?
    • needs a RODC - Samba AD as a read-only domain controller?
      • Requires=samba-ad-dc.service in the postfix unit ?
    • ACTION: [333] and [NTU] to experiment with Amazon EC2 VM, and set up basic mail server from scratch on it, and point a ucc subdomain at it, by the end of this weekend (Sunday 22/03/2020).

  • [FVP] Examples of how to recruit wheel members:

    • http://skynet.ie/guide/basic/introdoc.html
    • https://web.archive.org/web/20151214210546/http://www.csn.ul.ie/
      • was: http://www.csn.ul.ie/#recruit
    • ACTION: [MTL] to send this to committee to look into instead

  • [NTU] Samba AD familiarisation/monitoring/maintenance/updates/config-managed rebuilds

    • Extra DCs, RODCs? Samba AD domain controllers/read-only domain controllers
    • RODCs on mooneye (mail), mussel (shell, misc)
    • Deferring until after COVID outbreak
    • Anyone can poke at it in v.ucc.asn.au in the interim without breaking our prod. AD instance

  • [NTU] Demo: remote console access to the major servers?

    • [333] has action item above to test this

  • [NTU] Demo: restore files from offsite?

    • Not today [NTU]

  • [NTU] Communication of changes and proposed changes - Keep the tech/wheel group mailing lists up to date - Still preferred: email tech@ucc! - Good for public visibility: public archives - is https://wiki.ucc.asn.au/ChangeLog working? - not remembered by everyone - but good for public visibility - or can it be semi-automated? - where not recorded elsewhere?

  • [FVP] Regarding member data (specifically the membership register) and compliance with Privacy legislation, it would be worth discussing how we can be compliant and what an appropriate "privacy policy" would look like.

    • This is for committee to do
    • Delete this action item when we next have a wheel meeting [333]

  • [NTU] Hosting guidelines - this needs to end up clearly documented

    • Services
    • VMs
    • Physical
    • Encourage sprocket/winadmin enagement
    • Encourage secure practices a la SOE
      • Central logging for fail2ban?
        • Recommended logging config - document and/or ansible?
        • Also needed for servers
      • Uccroot keys?
      • Can/should AD accounts be used?
      • Install DebianPkg:unattended-upgrades , qemu-guest-agent
        • https://wiki.debian.org/UnattendedUpgrades
        • https://help.ubuntu.com/lts/serverguide/automatic-updates.html
      • https://www.xkcd.com/936/ , JtR , libpam-cracklib , libpam-pwquality

  • [NTU] Setup/offer cloud backup?

    • Think it should have a cost-per-gigabyte to it, but we can make it very cheap
    • [TEC] Sounds like an idea. Particularly the way that Storage costs are going. Also nice to have another way to "tempt" users into getting into ssh / CLIs / linux
    • Onsite, but backstopped by paid encrypted cloud storage:
    • https://rsync.net/products/attic.html USD$0.015/GB/month for actual hosted borgbackup servers
    • https://www.backblaze.com/b2/cloud-storage-pricing.html USD$0.005/GB/month raw storage, first 10GB gratis?
    • ACTION: Someone to move these links to wiki, along with hosting guidelines
    • Whoever does this gets to remove these items from the agenda :)

  • [TEC] ACTION: Investigate getting a GPU/compute server up and running

    • [BOB] i.e. Second hand Bitcoin miner seems like a good option

  • [BOB] Graham Bowland has rejoined

    • Was a former wheel member (purged 2019)
    • His account needs some hand-holding to be reactivated cleanly
    • Last logged in 2012
    • TODO: Move home directory out of wheel folder
    • TODO: Fix AD to point to the correct stuff
    • Can use samba-tool on samson (as root) to do a lot of this stuff

  • Account lockings are due

    • Current trasurer should find out who needs to be locked, and send email
    • Wheel to action based on email
    • Password resets:
      • Preferably in-person, but that may be difficult right now
      • Where possible: re-enabled accounts can be temporarily unlocked?
      • [BOB] Idea: Get requestor to video chat and hold up their ID to the cam. as verification step

  • [TEC] wants help with LDAP integration for Discord bot

    • Any takers?
    • Some not fully enthusiastic for this idea, as they may want to silo their social media
      • Good for self-hosted UCC services
    • Not a priority, but would be very much appriciated
    • Would want a new field in LDAP for discord ID, which is something like a uint64_t

ACTION ITEMS:

Wheel:

  • Annual account locking

  • Add a README.home-and-away or similar to both skeleton directories

  • LDAP/AD on uccmonitor

  • Minimal logging client and server setup - ansible: setup a new machine for central logging, journald/syslog forwarding

  • Check our Letsencrypt Certificate Transparency logs: Which ones do we need to automate?

    • mooneye upgrade

  • Figure out how to filter out configuration secrets from an etckeeper push mirror,

    • a la http://svn.ucc.asn.au:8080/rancid/ucc/configs/

  • Purchase a windows server licence

    • unsure if this can be done individually on connectingup.org

  • Contact [DBA] for a project that he'd like to do

  • Someone to move these links to wiki, along with hosting guidelines"

    • https://rsync.net/products/attic.html
      • USD$0.015/GB/month for actual hosted borgbackup servers
    • https://www.backblaze.com/b2/cloud-storage-pricing.html
      • USD$0.005/GB/month raw storage, first 10GB gratis?
    • Whoever does this gets to remove these items from the agenda :)

Teamwork action items: [333] and [NTU]:

  • Experiment with Amazon EC2 VM
  • Set up basic mail server from scratch on it
  • Point a ucc subdomain at it, by the end of this weekend (Sunday 22/03/2020).

[MTL], [NTU], and [TBB]:

  • to set up a regular chat to progress things

[NTU] and [MTL]:

  • Mail server hosting

[TEC] and [NTU]:

  • Automate email reminders
    • [TEC] will probably send email reminders every Monday during March until it?s automated

Solo action items: [MTL]:

  • Send examples of how to recruit wheel members to committee
    • http://skynet.ie/guide/basic/introdoc.html
    • https://web.archive.org/web/20151214210546/http://www.csn.ul.ie/
      • was: http://www.csn.ul.ie/#recruit
  • Drive an email discussion: re: DNS resolver split for authoritative
  • UniSFA
  • VPN things

[MPT]:

  • Parts buys: Make shopping list: network, rails, storage - on tech@ list

[LE@]:

  • Examine disc image on Monday and get back to UWA

[TRS]:

  • to set up bbb / online meetings

Meeting closed 17:05