At a previous job, where I was a senior sysadmin, I put together a proposal for a zero-trust approach to building a shipping code for both customer-facing apps and IaC work. I don’t know if it’s been adopted, but my current company is discussing something similar. Here’s a general outline, i’d be interested to get other folks thoughts on this approach for critical/regulated workloads. Note that this is based around Microsoft Azure, but the general methodology would be flexible, I think.
Introduction
With the growing complexity and evolving security threats, it’s important to consider a Zero-Trust model in DevOps processes. This proposal discusses a hypothetical Zero-Trust Azure DevOps environment that can ship code to production using Azure service principals and completely restricting human users from direct access to production systems.
Zero-Trust Philosophy
Zero-Trust is based on the principle of “never trust, always verify.” In a DevOps cycle, this means that no entity— internal or external—gets automatic access to resources. Authentication and authorization are mandatory for every interaction, even within our internal network.
Key Components
Azure DevOps
The CI/CD pipeline would be set up in Azure DevOps, which provides access control, change tracking, and workflow configs.
Azure Service Principals
Azure Service Principals would provide identities for our pipelines and applications. These credentials are securely stored and managed in Azure Key Vault.
Production Systems
Production systems are also hosted in Azure, with strict controls in place.
Implementation Strategy
Service Principals for Resource Access
Human users do not need direct access to our production environment; instead, Azure Service Principals are used. They handle all interactions with our Azure resources during the build and release pipelines. These service principals have only the permissions required for their specific role and are limited to specific scopes in Azure.
Multi-Factor Authentication and Conditional Access
Within Azure DevOps, consider implementing Multi-Factor Authentication (MFA) and conditional access policies, ensuring that only authorized personnel can make changes to the DevOps configurations.
Automated Testing and Code Scans
The pipeline includes rigorous automated tests, including security scans, to ensure that the code is both functional, in line with best practices, and free from vulnerabilities (as much as reasonably possible.
Secure Credential Storage
All sensitive information (API keys, database credentials client secrets), is stored securely in Azure Key Vault and only accessed by authorized service principals during the pipeline execution.
Logging and Monitoring
Azure Security Center/Defender and Sentinel are configured to provide real-time alerts and logging.
Immutable Infrastructure
Leverage DevOps pipelines to implement an immutable production infrastructure, meaning once a resource is deployed, it’s never modified. Instead, when changes are needed, the instance is replaced with a new instance upon the next release. This limits the risk of configuration drift.
Advantages
- Enhanced Security: Zero trust reduces the attack surface by eliminating implicit trust.
- Compliance: Assuming good management of the underlying infrastructure (sovereignty, encryption, etc.), helps to meet regulatory compliance requirements for data protection and access control.
- Auditability: Each interaction is logged, making it easier to monitor and audit activities.
- Operational Efficiency: Automated pipelines reduce the possibility of random error factor, improving reliability. Errors in automated processed can be identified, analyzed and fixed more quickly since the fix can be rolled out from one central source.
Challenges
Complexity and Cost
- Management Overhead: The Zero-Trust model inherently requires more management. This could include more time spent on configuring and monitoring pipelines, access controls, encryption, and logging.
- Cost: More resources may be needed for continuous monitoring, more advanced security tools, and additional Azure services. This could increase overall operating costs.
Operational Risks
- Employee Training: With a more complicated security architecture, there’s a need for additional in house training, or outside resources for adoption and overall solution design.
- Disaster Recovery and Rollback: The immutable infrastructure approach is secure but might complicate rollback strategies. In the event of a flawed release, rolling back to a previous state could become more complex and time-consuming if not configured properly
Security Concerns
- Insider Threats: While the system is designed to protect against external threats, it might still be vulnerable to internal threats, either malicious or accidental.
- Service Principal Compromise: If an attacker gains access to a service principal, they could potentially have the same level of access as the service principal itself, which could be highly privileged.
- Monitoring Blind Spots: Continuous monitoring is a critical part of this architecture, but it’s not foolproof. There may be blind spots and other configuration challenges with the monitoring system.
Dependencies
- Vendor Lock-in: The system heavily relies on Azure services. If there are service outages or simply as Azure features and costs change, the system would need to be flexible.
- Software Bugs and Vulnerabilities: Even though the system is designed to be secure, it’s still subject to vulnerabilities in the software that it uses, whether that’s in Azure itself or in other components of the DevOps pipeline.
Policy and Compliance
- Regulatory Changes: Compliance requirements change, and there’s a risk that future changes could mean significant changes to this architecture.
- Data Residency: Given that Azure is a global cloud provider, issues might arise regarding where data is stored and how it is transmitted internationally, meaning strict management would be required to stay in compliance with data sovereignty laws.
Auditing and Governance
- Auditing Complexity: While each interaction is logged, the sheer volume of logs could make auditing a monumental task.
- Governance: Enforcing policies consistently across such a complex environment can be challenging.
Josh Lambert 