The DevOps movement is a cultural shift toward collaboration and ownership. Collaboration among developers and users. Ownership of projects and features from idea to production. The ultimate result is higher quality and overall better experience for the users, all while creating a workplace where people want to work.
The Sarbanes-Oxley Act of 2002 (SOX) is a broad law that specifies certain requirements for public companies’ accounting and financial reporting. One element of the act addresses conflicts of interests. Consequently, the act requires separation of duties in key business processes. For IT, separation of duties means that developers who write code cannot also put that code into production. In contrast, DevOps strives to improve accountability by empowering individuals to see changes through from beginning to end. But, SOX forces a separation of duties in the delivery process. Furthermore, SOX requires that business processes be audited on a regular basis by an independent firm. By extension, IT processes must support the auditability requirement. Despite these challenges, it is still possible to achieve DevOps success in a SOX-compliant world.
DEVOPS PILLARS
According to Davis and Daniels in their book Effective DevOps, DevOps is built on four pillars: collaboration, tools, affinity, and scale. A successful DevOps environment in a SOX-compliant shop fulfills each pillar in a way that meets SOX requirements and still realizes the vision of the DevOps movement.
COLLABORATION
DevOps encourages collaboration among developers, users, and other team members who may not ordinarily work together. Senior developers may mentor junior contributors; the technical team and the stakeholders meet often to capture requirements / user stories and demonstrate functionality. The central themes in DevOps collaboration are still in effect in a SOX-compliant world. SOX-compliance requires separation of development and deployment; therefore SOX encourages knowledge transfer between developers and release managers. By integrating release managers into the project life-cycle, or, formally rotating development staff into the release management role, teams can collaborate and still adhere to SOX rules.
AFFINITY
Most SOX-compliant organizations do not deliver software as their primary business. IT enables business by delivering applications (bespoke or COTS) quickly and efficiently. Developers usually learn a great deal about the part of the business they support and provide immense value to the organization. At the same time, as a developer increases expertise (and thereby value) in one business area, there is a high switching cost to work in a different business area.
This high switching costs results in team silos. Silos force a developer to always work in the same area of the system, causing burnout. Additionally, siloed developers become key dependencies in which only one person knows a certain system area.
DevOps culture discourages silos. Elimination of silos prevents burnout of key individuals and mitigates risk in the event a key developer leaves the team. DevOps encourages continuous learning and team rotation, preventing team members from getting stuck in a single area of the system.
Closely related to silos is Hero Culture. Hero Culture is a situation in which the team is organized around one or two individuals who act as heroes, sought after to solve every problem. Hero Culture has the same issues as silos, but has the added problem of undermining team potential.
DevOps seeks to correct these issues by encouraging methodologies and practices that help communication. Pair programming, Agile development, SCRUM project management are themes that accompany DevOps. SOX-compliance is not necessarily at odds with such ideas, but team role changes often must be made “official”. For SOX-compliance, team members must be granted specific rights to change production in certain roles (e.g. release management) and those rights must be explicitly revoked in other roles (e.g. development).
SCALE
Scaling can mean different things in different situations. For startups, scaling generally means expanding the customer base and the staff to accommodate the growth. For an established enterprise that packs on transactions daily, scaling can mean keeping staff and SLAs constant event as the transaction volume accumulates.
Essential elements that facilitate scale include efficient onboarding, automation (including automated regression testing), and system monitoring & alerting. SOX does not call out these elements explicitly, but they work together to deliver change while managing risk.
Using virtual machines as the primary development platform ensures uniform tooling and keeps sensitive data within the network.
Team members should be encouraged and guided to automate everything that is done more than once. Standardize on a limited number of scripting languages. Demo and document their use.
SOX-compliant organizations tend to have a lot of financial risk tied up in their IT systems. Financial risk drives service level agreements (SLAs). SLAs cover outage limits (outage detection and time to recover), and system performance (throughput and/or response times). Depending upon demands of the SLAs, system monitoring and alerting can be complex.
TOOLS
DevOps does not require specific tools, but certain tools are commonly associated with DevOps culture. “DevOps” tools promote the other three pillars by encouraging collaboration, preventing silos, and helping the team to scale. Many tools are open source. In SOX-compliant situations, open source tools and libraries should be used with care and scrutiny. Most SOX organizations do not build software as their primary business; the IT function exists to serve the business. Also, IT is viewed as a business cost center and missteps can cause a great deal of harm to IT’s internal reputation and exacerbate the cost to the bottom line. When selecting tools, pay attention to licensing and available support. Certain licenses (e.g. GPL, LGPL) place restrictions on how software can be used for commercial purposes. Other licenses (e.g. Apache, MIT) are more permissive.
Additionally, tools (and libraries) must a have a support policy. Support must be available to at least handle production outages and security updates. In the absence of support, the organization takes on the full responsibility of keeping software up to date with security patches and dealing with any production issues caused by defects in open source libraries. That may not be a showstopper to some businesses, but most do not have the resources to self-support third-party tools and libraries.
SOX-compliant organizations must subject themselves to regular third-party audits. Auditors must be able to relate all production changes to a specific person at an exact time. From that information, auditors must confirm that the person who made the change to production did not author the change, and that the change received the proper specification, testing and approval.
Any tool used to deploy production changes in a SOX-compliant shop must enable auditing through authentication, authorization, and logging. The tool must record who is using it. The tool must permit only certain users to perform critical functions (i.e. the functions that actually make the change to production). The tool must log all actions and the log itself must be secure (i.e. written to a filesystem / database in which only certain users have write access).
Much of the SOX auditability requirement thwarts the DevOps objective to promote full change ownership from idea to delivery. IT shops solved this problem by creating environments that replicate production. By giving developers full access to a production-like environment, they can deploy, test, and demo their changes without drawing ire from auditors. Since release managers must perform the final task of delivering the change to production, the change must be documented and rehearsed so that it is delivered smoothly to production. Each change minimally goes through two official deployments prior to production – once as the developer readies the change for testing and demonstration, and again as the release manager tests the deployment itself and absorbs the change details.
CONCLUSION
At first, DevOps seems at odds with a SOX-compliant world. A deeper look reveals that DevOps can improve the drudgery and stress of a SOX-compliant IT shop. You can be the change agent to move the organization toward a DevOps culture.
FURTHER READING
1) Effective DevOps by Jennifer Davis and Ryn Daniels