Security, governance, and compliance for cloud-scale analytics

When planning cloud-scale analytics architecture, pay special attention to ensure that the architecture is robust and secure. This article addresses security, compliance, and governance design criteria for enterprise-scale cloud-scale analytics. This article also discusses design recommendations and best practices for deployment of an cloud-scale analytics on Azure. Review enterprise-scale security governance and compliance to fully prepare for governance of an enterprise solution.

Cloud solutions initially hosted single, relatively isolated applications. As the benefits of cloud solutions became clear, larger-scale workloads were hosted in the cloud, such as SAP on Azure. So it became vital to address the security, reliability, performance, and cost of regional deployments throughout the lifecycle of cloud services.

The vision for cloud-scale analytics landing zone security, compliance, and governance on Azure is to provide tools and processes that help you minimize risk and make effective decisions. The Azure landing zones define security governance and compliance roles and responsibilities.

Cloud-scale analytics pattern relies on several security features that can be enabled in Azure. These features include encryption, role-based access control, access control lists, and networking restrictions.

Security design recommendations

Both Microsoft and customers share responsibility for security. For accepted security guidance, refer to Cybersecurity best practices by the Center for Internet Security. The following sections are security design recommendations.

Data-at-rest encryption

Data-at-rest encryption refers to the encryption of data as it persists in storage, and addresses the security risks related to direct physical access of storage media. Dar is a critical security control since the underlying data is unrecoverable and can't be changed without its decryption key. Dar is an important layer in the defense-in-depth strategy of Microsoft datacenters. Often, there are compliance and governance reasons to deploy data-at-rest encryption.

Several Azure services support data-at-rest encryption, including Azure Storage and Azure SQL databases. Although common concepts and models influence the design of Azure services, each service can apply data-at-rest encryption at different stack layers or have different encryption requirements.

Secure data in transit

Data is in transit or in flight when it moves from one location to another. This can be internally, on-premises or within Azure, or externally, such as across the internet to an end user. Azure offers several mechanisms, including encryption, to keep data private in transit. These mechanisms include:

• Communication through VPNs using IPsec/IKE encryption.
• Transport Layer Security (TLS) 1.2 or later used by Azure components such as Azure Application Gateway or Azure Front Door.
• Protocols available on Azure Virtual Machines, such as Windows IPsec or SMB.

Encryption using MACsec (media access control security), an IEEE standard at the data-link layer, is automatically enabled for all Azure traffic between Azure datacenters. This encryption ensures customer data confidentiality and integrity. For more information, see Azure customer data protection.

Manage keys and secrets

To control and manage disk encryption keys and secrets for cloud-scale analytics, use Azure Key Vault. Key Vault has capabilities for provisioning and managing SSL/TLS certificates. You can also protect secrets with hardware security modules (HSMs).

Microsoft Defender for Cloud

Microsoft Defender for Cloud provides security alerts and advanced threat protection for virtual machines, SQL databases, containers, web applications, virtual networks, and more.

When you enable Defender for Cloud from the pricing and settings area, the following Microsoft Defender plans are enabled simultaneously and provide comprehensive defenses for the compute, data, and service layers of your environment:

• Microsoft Defender for servers
• Microsoft Defender for App Service
• Microsoft Defender for Storage
• Microsoft Defender for SQL
• Microsoft Defender for Kubernetes
• Microsoft Defender for container registries
• Microsoft Defender for Key Vault
• Microsoft Defender for Resource Manager
• Microsoft Defender for DNS

These plans are explained separately in the Defender for Cloud documentation.

Microsoft Defender for Identity

Microsoft Defender for Identity is part of the advanced data security offering, which is a unified package for advanced security capabilities. Microsoft Defender for Identity can be accessed and managed via the Azure portal.

Enable Microsoft Sentinel

Microsoft Sentinel is a scalable, cloud-native, security information event management (SIEM), and security orchestration automated response (SOAR) solution. Microsoft Sentinel delivers intelligent security analytics and threat intelligence across the enterprise, providing a single solution for alert detection, threat visibility, proactive hunting, and threat response.

Networking

Cloud-scale analytics prescribed view is to use Azure private endpoints for all PaaS services and not use public IPs for all infrastructure as a service (IaaS) services. For more information, see Cloud-scale analytics networking.

Compliance and governance design recommendations

Azure Advisor helps you get a consolidated view across your Azure subscriptions. Consult Azure Advisor for reliability, resiliency, security, performance, operational excellence, and cost recommendations. The following sections are compliance and governance design recommendations.

Use Azure Policy

Azure Policy helps enforce organizational standards and assess compliance at scale. Through its compliance dashboard, it provides an aggregated view of the overall state of the environment, with the ability to drill down into individual resources or policies.

Azure Policy helps bring your resources into compliance through bulk remediation of existing resources and automatic remediation of new resources. Several built-in policies are available, for example to restrict the location of new resources, require a tag and its value on resources, create a VM using a managed disk, or enforce naming policies.

Automate deployments

You can save time and reduce errors by automating deployments. Reduce the deployment complexity of end-to-end data landing zones and data applications (which create data products) by creating reuseable code templates. This minimizes the time to deploy or redeploy solutions. For more information, see Understand DevOps automation for the cloud-scale analytics in Azure

Lock resources for production workloads

Create required core data management and data landing zone Azure resources at the start of your project. When all additions, moves, and changes are finished, and the Azure deployment is operational, lock all resources. Then, only an administrator can unlock or modify resources, such as a data catalog. For more information, see Lock resources to prevent unexpected changes.

Implement role-based access control

You can customize role-based access control (RBAC) on Azure subscriptions to manage who has access to Azure resources, what they can do with those resources, and what areas they have access to. For example, you can allow team members to deploy core assets to a data landing zone, but prevent them from altering any of the network components.

Compliance and governance scenarios

The following recommendations apply to various compliance and governance scenarios. These scenarios represent a cost-effective and scalable solution.

Next steps

Azure policies for cloud-scale analytics