Azure DevOps Landing Zone

Uncategorized
by csaba

In the context of cloud infrastructure and corporate usage, a Landing Zone refers to an environment or a set of guidelines, best practices, and resources that allow organisations to set up their cloud infrastructure securely and efficiently. It serves as a foundation for deploying services and resources in a cloud environment. The concept is especially relevant when migrating large-scale, complex IT workloads in the cloud.

While landing zones have many aspects, I will try to collect them into 4, and elabore on one of them.

  1. Security and Compliance: This may include configurations for identity and access management, data encryption, logging and monitoring, and other security mechanisms. Identity and Access management may fall into this category.
  2. Networking: Sets up foundational networking components like Virtual Private Clouds (VPCs), subnets, and connectivity between on-premises data centers and the cloud. Hybrid connectivity is a large topic, that falls under this category.
  3. Resource Organization: It offers strategies for setting up and organizing cloud resources, possibly by using folders, resource groups, or other organizational constructs. It establishes tools and boundaries to monitor and manage costs, usage, and performance.
  4. Automation and DevOps: Often, landing zones include Infrastructure as Code (IaC) templates and automation scripts to ensure consistent and repeatable deployments. In the next chapters, we will elaborate on this DevOps concept.

Microsoft Azure has landing zone description on its own, that provide guidelines, best practices, and resources for setting up an environment for the deployment of services and workloads. Again, what will be proposed here does not alter, but rather augment and detail the Microsoft LZ principle.

GitOps as Landing zone enhancement

This GitLab demonstrates a powerful concept of cooperation between

The concept is geared towards keeping all your project as code in GitHub. Apart from your proper source code this incompasses

Lets take each of these components one by one. We will end up with following actors, and I will explain each of them in subsequent chapters.

The bootstrap script

As we will attempt to move as many projects under this umbrella as possible, the setting up of new projects, or the migration of existing projects should be as straightforward as possible. The following python script prepares the LZ for a new project.

Bootstrapping consists of linking GitHub (line 12) and TFE (line 35) together on the behalf of a project. GitHub forks a seed project, that is an adjacent repository, and is used as a blueprint for all new projects. TFE sets up a new workspace, which is a cloud managed environment to execute the terraform scripts from a central managed location. 

parser = argparse.ArgumentParser()
parser.add_argument("-i", "--projectId", required=True)
parser.add_argument("-n", "--projectName", required=True)
parser.add_argument("-g", "--ghOrganization", required=True)
parser.add_argument("-t", "--tfeOrganization", required=True)
args = parser.parse_args()

logging.info(f"Args: {args}")
now = datetime.now() # current date and time
current_timestamp = now.strftime("%m/%d/%Y, %H:%M:%S")
 
def setupGithub(args, current_timestamp):
    logging.info(f"Setting up the github connection")
    github = Github(config("GH_PAT"))

    # set the marker
    logging.info(f"Adding repo description for organization {args.ghOrganization}")
    projectOrg = github.get_organization(args.ghOrganization)
    projectOrg.edit(description=f"Bootstrapped by LZF on {current_timestamp}")

    # fork the lzf-seed into this new org
    # find the reference to the repo to copy
    lzfSeedRepo = github.get_repo("lzf-azure/lzf-seed")
    projectLzfRepoName = f"lzf-{args.projectId}-{args.projectName}"
    logging.info(f"will fork repository: {lzfSeedRepo.full_name} into : {args.ghOrganization}/{projectLzfRepoName}")
    # Solution: use clone and push copy instead 
    projectLzfRepo = lzfSeedRepo.create_fork(organization=args.ghOrganization)

    # set the properties of the new repo
    projectLzfRepo.edit(name=projectLzfRepoName)
    projectLzfRepo.edit(delete_branch_on_merge=True)
    projectLzfRepo.enable_vulnerability_alert()

def setupTfe(args, current_timestamp):
    logging.info("Logging in to TFE org: %s", args.tfeOrganization)
    client = pyterprise.Client()
    client.init(config("TFE_USER_TOKEN"), url='https://app.terraform.io')
    org = client.set_organization(id=args.tfeOrganization)

    logging.info("Existing TFE Workspaces:  %s", len(org.list_workspaces()))
    projectLzfRepoName = f"lzf-{args.projectId}-{args.projectName}"

    vcs_options = {
        "identifier": f"{args.ghOrganization}/{projectLzfRepoName}",
        # "identifier": "abaxsoraszem/conntest",
        "oauth-token-id": config("TFE_GH_OAUTH_TOKEN_ID"),
        "branch": "main"
    }

    logging.info(f"Creating workspace for VCS {vcs_options}")
    org.create_workspace(name=projectLzfRepoName,
                        vcs_repo=vcs_options,
                        auto_apply=False,
                        queue_all_runs=False,
                        working_directory='/',
                        trigger_prefixes=['/'])
    workspace = org.get_workspace(projectLzfRepoName)

    logging.info("Adding variables to access github from terraform provider")
    workspace.create_variable(key='GITHUB_TOKEN', value=config("GH_PAT"), sensitive=True, category='env')
    workspace.create_variable(key='GITHUB_OWNER', value=args.ghOrganization, sensitive=False, category='env')
    workspace.create_variable(key='projectId', value=args.projectId, sensitive=False, category='terraform')  
    workspace.create_variable(key='projectName', value=args.projectName, sensitive=False, category='terraform')  

    logging.info("Workspace runs: %s", len(workspace.list_runs()))

    logging.info("Running intial TFE apply on workspace %s. Dont forget to apply manually", projectLzfRepoName)
    workspace.run()

Manual setup

This script only attempts to demonstrate the possibilities of enhancing the LZ Concept, and it leaves couple of steps for manual intervention. These are

GitHub

  • manualy create GH Organization for the project (testt-projectid-name)
  • export GH_ORGANIZATION=testt-projectid-name
  • manually create the Github GH_PAT with sufficient OAUTH rights in Settings -> Developer Settings -> Personal Access Token
  • export GH_PAT=ghp_lRydJ...

TFE

  • set up your TFE user token export TFE_USER_TOKEN=123.atlasv1.456
  • create a new TFE Organization, eg lzf-test-123 This will be the organization, where TFE will be hosting the workspaces for the GH repositories
  • create the Terraform Enterprise Organization Token TFE_ORG_TOKEN as Organization -> Settings -> API Token -> Create Organization Token. This token will allow the python script to access the TFE API on behalf of the TFE Organization
  • export TFE_ORG_TOKEN=asdf.gsgf.gad
  • create TFE connection to GH (OAUTH Token): TFE -> Organization Settings -> Providers -> Add VCS Provider -> github.com(custom). You will need to also create the GH Application to which it is linked, and copy the Client ID and Client Secret back to TFE. Ultimately, GH will issue and OAUTH token for TFE, which will be referenced by TFE_GH_OAUTH_TOKEN_ID in the bootstrap script. At this step, you will need to grant access to the GH organization created above. Note: if the repo is forked, then the upstream organization must also grant access.
  • export TFE_GH_OAUTH_TOKEN_ID=ot-...

For further details on how to run this, refer to the bootstrap repo of the solution.