Introduction
One of IT’s favorite new buzzwords is “Infrastructure as Code” or “IaC”. The definitions of IaC can vary, but they all refer to some variation of a declarative language for defining Infrastructure components, and characteristics.
Cloud Management and Automation tools have been embracing this new mechanism for several years, and of course vRealize Automation has also done so with vRealize Automation 8.x and vRealize Automation Cloud.
Here is a very simple snippet of IaC:
formatVersion: 1
inputs: {}
resources:
Cloud_Machine_1:
type: Cloud.Machine
properties:
image: windows
flavor: Small-windows
constraints:
- tag: 'env:vsphere'
remoteAccess:
authentication: usernamePassword
username: Administrator
password: VMware1!
cloudConfig: |
#cloud-config
write_files:
content: Cloudbase-Init test
path: C:\test.txt
set_hostname: WindowsVM
networks:
- network: '${resource.NSX.id}'
NSX:
type: Cloud.NSX.Network
properties:
networkType: routed
constraints:
- tag: 'net:on-demand'
The first and most prominent functions for IaC within vRealize Automation (vRA) are within Cloud Assembly, with its YAML declarative format for blueprints, version and release control and integrations with Git.
But it is possible that the most value from IaC within vRealize Automation can be “realized” (no pun intended) in the brand new, totally revised versions of Code Stream (included with vRA Enterprise and all vRA Cloud).
It is within Code Stream where the full potential of IaC can truly be exploited. The reason is simple: it permits infrastructure admins – of ANY infrastructure platform; on-prem or public cloud and all variations – to leverage the very same techniques that Application developers have been leveraging for years with the revolution in Continuous Integration, Continuous Delivery, Agile Development models ….. colloquially, DevOPs.
The idea is simple: As you define infrastructure, you do so in a collaborative, continuous development cycle, using heterogenous toolsets from multiple commercial and open-source providers. As you develop and modify infrastructure components destined for your Service Catalogs and ultimately production, you leverage what is commonly regarded as the core component of CI/CD: the pipeline.
Just as agile application developers use pipelines to stage their application code from a developer’s desktop to testing environments and finally production. Infrastructure as IaC can be managed in the same way.
vRA Code Stream offers a very powerful Pipeline Management capability for BOTH applications and IaC. Code Stream has builtin integrations with most commonly use development and testing tools.
However, as organizations adopt IaC and Code Stream with vRA 8.x, there are choices and considerations for how to architect and IaC Pipeline environment.
This article will outline many of the architectural options available, with considerations for how IaC operations can influence your vRA 8.x design.*
*From this point forward, the remainder of this article will focus on the on-prem form factor for vRA and vRA Cloud is a SaaS service and does not require the same design decisions for organizations
General Architecture Considerations.
The default preference for vRA 8.x architectures for large enterprises is as few instances of vRA 8.x “air-gapped” instances as possible. Ideally, one “live” instance is the optimal architecture. However, it is expected that most organizations will seek some level of redundancy for practical operational purposes:
1. Disaster Recovery
2. “Lab” or “Sandbox” instances where pure experimentation can take place without impacting operational instances of vRA
NOTE: In this section, this article will purposely avoid use of terms like “lab”, “dev” and “production” to avoid confusion with the same terms used for managed workloads (under management BY vRA), vs. vRA itself as an application. The term “operational” will be used to describe non-experimental / sandbox vRA instances. DR instances, for the purposes of this discussion will not be considered “operational”.
Potential Reasons for Multiple Operational Instances of vRA
There are three possible reasons for requiring multiple (“air gapped”) operational instances of vRA 8.x
1. Customer’s requirements exceed configuration maximums as certified and published by VMware engineering for vRA 8.x best practices.
2. Geographical considerations: IE WAN limitations
3. Role-based Access Control (RBAC) or security Requirements
1. Configuration Maximums:
Please refer to the following document for configuration capacity of vRA 8.x. These include measurements for maximum concurrent operations, maximum workloads under management, etc.
2. Geographical (WAN) Limitations
In contrast to earlier versions of vRA (version 7.x and earlier), vRA 8 is a completely new and distinct platform. vRA 7 reference points are not relevant. vRA 8 architecture is completely different that prior releases of vRA and based on entirely different, more modern technology. For example, there is no counterpart in vRA 8 for vRA 7 node elements such as “Distributed Execution Modules” (DEMs). There is no “Orchestrator DEM” or “Worker DEM”. There are no “management agents” necessary to connect to Cloud Accounts (fka “Endpoints” in vRA 7.x).
All “Cloud Account” connections are API-based and designed to function in a modern, multi-cloud environment. This includes all supported public cloud platforms (Azure, AWS, Google Cloud Platform) and vSphere endpoints.
However, there is a requirement for reliable and predictable wide-area network connectivity and network latency considerations.
The current, documented requirement for network latency to any Cloud Account under management by vRA 8 is under 500 milliseconds (ms). Latencies over 500ms will result in unpredictable behaviors and potential intermittent outages.
3. Role-based Access Control (RBAC)
vRA 8.x is designed to function as a single instance platform with roles and management to be fashioned within the management constructs within vRA (Projects, Tenants, Cloud Zones, etc).
However, as the RBAC models of vRA 8.x expand, some customer requirements for the current release of vRA 8.x may not entirely work for certain customers’ desired functions and roles.
NOTE: At the time of this writing the current release of vRA 8.x is vRealize Automation 8.1. Also, vRA Cloud, the SaaS implementation of vRealize Automation is on a separate release cycle. More granular RBAC functions are to be implemented in upcoming vRA releases
Projects, Tenants or vRA Instances as Access Boundaries
The default optimal vRA 8.x architecture is to NOT require multiple instances of vRA 8.x to meet requirements. In order of “preference”, the mechanisms to enforce RBAC would be met by implementation of these constructs:
1. Projects – Projects are the most lightweight and flexible mechanism for manageable RBAC. Projects are designed to contain distinct groups of users, assigned access to distinct cloud platforms (Cloud Accounts), and distinct catalog content (Blueprints). Projects have well defined roles: Managers who are delegated governance over use of resources by all other consumers; and Users, who are the consumer “end users”. There is also a global “Read Only” role which is more restricted to “Users” in that they can view resources but cannot alter them or request deployments. Projects are relatively simple to provision quickly, even on-demand.
2. Tenants – Tenants (or Orgs) are supported to segregate business units who are intended to share physical resources (Cloud Accounts, and Cloud Zones), content (Blueprints) but are not intended to be aware of other business units’ access. Most often, Orgs are entirely different Businesses (EG: “Coke” vs. “Pepsi”) sharing common, but segmented infrastructure. Org Admins can create Projects and assign roles to groups within those Orgs. In some cases distinct groups within ONE organization or business may need to be
treated as different entities (EG: “Transportation” and “Logistics”) Configuration of Tenants requires multiple steps and careful planning at several points within vRA 8.x, and vRealize Lifecycle Manager. Orgs and Tenants are not practical for “on demand” operations. (Typical turnaround is multiple days)
3· Separate Physical Instances (Air-gap) – some customers may require highly segmented and granular RBAC controls between groups of users and infrastructure resources that cannot be provided by Projects or Orgs. For example, certain organizations may require STRICT control of “Production” resources that are tightly restricted to distinct “low access” consumers and a very small number of highly privileged administrative users. In such instances one or more additional instances of Operational vRA 8.x instances may be necessary. Clearly, this requirement has administrative and infrastructure costs of its own: -An increased physical footprint for vRA 8.x -Administrative overhead -Shared content is possible but requires carefully designed integration with other tools (EG: vRealize LifeCycle Manager, and/or Git, etc)
For each of the models above, an administrative model should be considered and evaluated. The remainder of this document is a compilation of potential use cases and administrative model for an enterprise implementation of vRA 8.x and its core services (Cloud Assembly, Service Broker, Code Stream and vRealize Orchestrator).
In addition, inclusion of third-party resources may be recommended as examples.
vRealize Automation IaC DevOps Deployment Models
There is no single architecture for an Infrastructure as Code architectural model that works for every customer or every use case. There will be advantages of each model and disadvantages to the same. The final result will depend on the absolute requirements of the organization and its constituent user groups.
However, certain tools and constructs are assumed to be in use in most examples. Since code release repositories and code editing tools are widely available and in use for application code, it makes sense to leverage those tools to IaC code as well.
The internal construct of code repositories, such as Git (Github, GitLab, etc) will offer constructs that can be mapped to appropriate constructs in vRA, in different combinations.
Below is such an example. Each IaC blueprint repository in Git is mapped to a Project in vRealize Automation. In turn, each Project represents a Stage in the continuous deployment process. Each Project requires a compute/storage/network environment for the staging functions to be performed, from development through production. Those environments are represented as Cloud Zones in vRA 8. Most, if not all of the user authoring tasks in this example are done using available text/code editing tools. Any text editor can be used, but tools such as VS Code have direct useful integrations to code repositories which are useful.
Finally, the example below depicts an IaC Pipeline model that has the final stage (“Prod”) assigned to a separate, dedicated instance of vRA 8. This is not required. This model offers certain advantages, and disadvantages, which will be reviewed compared to other examples:
Here is a different representation of the model depicted above, with user roles and functions performed at each stage added. This model, like the example above, also represents a “Prod” environment with its own, dedicated instance of vRA 8. Content, between the “non-prod” and “prod” environments can be synchronized using any number of methods, but this example assumes that Git will be the content backbone and as content changes pass from stage to stage, they will be synchronized to vRA 8.
This model has certain advantages, and disadvantages:
Advantages:
Production workloads are tightly controlled within a dedicated RBAC model
Content changes are strictly controlled, using Git replication as a security-access barrier. (IE Users are typically prevented from direct editing of content)
Accidental, or authorized changes are less likely as they must originate first outside of the “Prod” instance (ie no direct editing)
User Access (consumers) can be easily regulated using Project Roles
Disadvantages:
The primary disadvantage of this model is the increased resource usage of vRA 8 (more compute, storage etc)
Configuration management across instances will require additional management overhead.
The three-stage model depicted above, can be implemented in a single instance of vRA.
In this model, instead of a separate instance of vRA, “Prod” is represented as a Project within vRA. Alternatively, Tenants can be used to separate “Prod” and “Non Prod” environments. However, multiple Tenants also require detailed configurations to Identity Management, DNS etc, which add additional admin overhead.
Advantages:
The obvious advantage to this model is that it is the smallest physical footprint for vRA possible.
There is no need for additional infrastructure, configuration management etc.
The RBAC model within Projects operates as the security barrier between stages.
Disadvantages:
Some customers may want or require higher levels of guaranteed security that Project RBAC roles and privileges cannot offer in this model.
Also, if the environment grows, this model may be harder to scale as capacity demands increase, as all constructs were designed from the outset with a single vRA instance in mind.
Other variants:
These models can be conceptualized in many combinations. In the model below the “Staging” stage was removed (for the sake of simplicity only) and the “Prod” environment has been separated into “Blue” and “Green” environments.
In this instance, it is likely both Blue and Green would be treated as “Production” as the Blue/Green deployment model is design for always having a “fall back” model in production in case changes to one production environment fail.
This model is based entirely on a single instance of vRA, again using Projects as the primary security construct.
However, if “air gapped” security is required, “Production” may be separated in to a second vRA instance (even a third, if “Blue” and “Green” are require their own strict security barriers).
This depicts a Blue/Green model in a single instance of vRA 8.x
However, in a multiple Instance model where “Prod” is treated separately, it is likely that both “Blue” and “Green” environments would be classified as “Prod” so such a model would likely appear as such:
The Role of Code Stream and Pipelines
In each of the depicted models, the vRA construct that would procedurally bind these environments together would be Pipelines. In vRA 8.x, this would be the role of Code Stream.
The detailed steps within a Pipeline will be very unique to the release process of each customer, the steps of governance required, the degree of automation within the release cycle and the role of specific groups and individuals.
However, certain core structures will be very common at a relatively high level.
Below is an example of a Multi-Stage release process across THREE stages of a release cycles. This is only an example, but the three stages are:
“Development” – Assigned stage for Security penetration testing
“Staging” – Assigned stage for Application functional testing
“Production” – Final release to Catalog for deployment
NOTE: The order of these stages is purely notional, but in the stages depicted and the order of execution. Actual sequencing is the decision of the IaC development team. Also, the level of automation within each task within each stage will vary, depending on the organization’s processes and available tools.
An example of such a release cycle is depicted below:
In the above scenario: EACH STAGE of the release cycle is handled by a separate pipeline. (Three pipelines).
It is possible that the entire process depicted above could be achieved in a single pipeline.
There are certain advantages, however, of executing such a process in more “modular” pipelines:
Changes to processes in one stage need not be dependent on other stages
Visibility to process execution can be limited to those whose roles require it
Complexity can be isolated within stages
Modular pipelines can be reused across multiple release cycles that may be different at the macro level.
The above process does not necessarily use a MASTER pipeline executing modular, nested pipelines. In this case, there could merely be three separate pipelines, executed in sequence.
This may or may not make failures at certain tasks more difficult to isolate from a management perspective.
An alternate approach could be to utilize a FOURTH, “Master” pipeline that orchestrates the entire process as a whole. Success/Failure at the “Master” pipeline will be tracked, AND task success/failure and the “nested” (or “modular” or “component”) pipelines can be tracked separately at the same time.
Effect of vRA Instances on Pipeline structure:
Examples of Release Processes, such as those depicted above can be implemented on Single OR Multiple Instances of vRA 8.x.
Design and execution of such a process in a single instance would likely be simpler, since all tasks and stages are contained in a single instance of Code Stream.
However, if Stage segregation is required, the same process can still be achieved. Pipelines in separate instances of Code Stream would only be able to execute on environments managed by that vRA 8.x instance. But, through API calls, a Master Pipeline in one instance of vRA 8.x/Code Stream can call and execute a Pipeline in another vRA 8.x instance.
Summary:
As you can see, there are many ways you can architect an IaC development environment. Your specific design will vary according to your operational needs. As a general rule, or preferance simpler, lightweight architectural models are preferable for the reasons cited earlier.
If your organization requires more specific recommendations and detailed operational recommendations, you can consult VMware Professional Services or services from VMware’s network of solution partners.