System Configuration Validation

Whether for regulatory compliance or to minimize drift, it is always a good idea to monitor key aspects of your server’s system configuration over time. Unfortunately, even virtual machines deployed from the exact same template can sometimes act differently. Yes, this is IT, and s**it does happen, blame Murphy and his law. There is also constant patching, library versions that are always evolving, keeping constancy over a large number of servers over time is a real challenge.

There seem to be two distinct philosophies on how to achieve this state. Some believe that it is best to use configuration management tools such as Chef, Puppet, or Salt to periodically overwrite current settings and reset the system's configuration to its initial state.

I personally believe that this behavior is risky, automatically changing the configuration of a production system without proper due diligence is unwise. I much prefer the "audit and alert" approach, check each configuration item against what is expected and alert or open a ticket when a discrepancy is detected.

After much research, I settled on Goss, for "Quick and Easy server validation". Goss truly lives-up to its promise of being "quick and easy", just download the self-contained binary for Linux, Windows, and macOS, and you're ready to do some validation.

The syntax is very straightforward and well documented, and allows you to check for a variety of items, such as packages being installed or not, services enabled and running, the content of files, etc. The only negative I found was the backward logic of some of the error messages when a required item is missing. Otherwise a solid tool, highly recommended.

I ended up building layers of Goss validation rules, a baseline that all systems should have, the baseline cybersecurity compliance layer, and then added layers based on the type of servers and their function, as well as the required parameters (see Tuning Linux Parameters). Rules were pulled out of a GitHub repository "live" by each server, and errors were logged as issues in another GitHub repository, maybe I'll post the full code someday.