Development

We are a Test Driven Development (TDD) shop. All application development takes place within the auspices of a/the most popular/supported testing framework within that development language (eg pytest for Python; robocop for Ruby).

All software we deliver is packaged using BastionLinux/RPM. This ensures (i) dependency management; (ii) versioning/lifecycle management; (iii) provenance of every file artefact on a system.

The actual configuration of this delivered software is mostly via our Chef infrastructure where we’ve written a complex suite over many years to deploy hardened, enterprise-ready systems.

Frameworks

We never start from scratch! Everything we do is delivered within the context of an extensible framework. We do this (i) to jumpstart the project via that ecosystem of features; (ii) ensure a community of technical skills and documentation for the customer; (iii) the community moves faster than the individual - the application lifecycle management will prove superior in the long run.

Frameworks we use for development include Django, Flask, Plone, pytest, Sphinx, Jenkins.

For monitoring Grafana, Prometheus; for orchestration Ansible, Teraform, Chef, Knife.

Testing

Unfortunately cross-platform web development still appears to be a thing; to test HTML-based web services across browsers/devices we use BrowserStack.

This too proves slightly non-trivial. Selenium is/has migrated to it’s own manager to locally install browsers and drivers. This is (i) an anathema to RPM-based installations; (ii) while IOS/Ubuntu appear well-supported; other Linuxes are not so much.

Selenium Manager is the prescribed rust-based suite which is baked into python3-selenium to do this; we’ve written our own python-selenium-manager which uses RPM to download all the necessary drivers and browsers for a fully functional local Selenium.

We use RobotFramework and its RIDE integrated development environment to run and edit browser-based testing. We integrate this with the vanilla RobotFramework/Jenkins plugin for CI/CD - giving a pretty GUI to step though failed tests et al.

Generically, we use Python/Tox as our CI/CD driver (across programming languages) to set up environments and trigger tests. Preferably tests should support JUnit/XML as Jenkins jobs are well-supported for this.

Obviously this is restricted to what is runnable within our Linux/X86 ecosystem. For other browsers/devices we use BrowserStack - which is a remote service with dedicated native hardware and emulators across a range of mobile and other OS’s to assure web properties will render across many consumer/customer devices in the wild.

Assurance

This is the process of ensuring systems and services are up and running, listening on expected ports, and are basically configured as expected.

We have written a large Inspec suite which does the real-time scanning and reporting of this. inspec is a programming and DSL environment allowing you to describe components of your system - and then make assertions about the validity of what is described. Failures need to be investigated and corrected.

This is an ongoing process where incidents are investigated and codified within Inspec. If the solution is a configuration change - then we update our Chef; if it is software - then we correct the bug and release a new RPM. The next Chef/orchestration on the host will deploy the new software version and/or write new configuration and reload/restart affected service(s).

Many vendors and cloud providers ship Inspec profiles for their systems. We have RPM bundles for these; and utilise them within our own Inspec.

Scheduling

Much of the above is periodic in nature; we need to understand when an anomaly began to occur. We have two mechanisms for this: Jenkins and Airflow.

Our Jenkins provides a fantastic dashboard for RAG. Our Airflow does this too; but its programmatic/workflow processing also provides an opportunity to automate remediations (if indeed they exist).