MOR-PROC-025 Version 1 Last Review Date: Nov, 2024
Azure Specific Resource Corrupted
1. Failure Scenario
This recovery plan relates to the following Failure Scenarios:
| Failure Scenario ID | Title | Link |
|---|---|---|
| 5 | Azure Primary Region Down | Link |
When an Azure resource is corrupted, it refers to a situation where the resource, such as a Virtual Machine (VM), database, storage account, or another Azure service, has become unusable or is experiencing significant issues, such as data corruption, configuration failure, or operational malfunction
Identify the Corruption
Check the Azure Portal for any alerts or error messages that indicate corruption or issues with the resource. Symptoms: Notice issues such as failure to start a VM, inability to access data, connectivity problems, or application crashes.
Error Messages: Check the Azure Portal for any alerts or error messages that indicate corruption or issues with the resource. Symptoms: Notice issues such as failure to start a VM, inability to access data, connectivity problems, or application crashes.
Azure Activity Logs: Review the Activity Logs to see if any changes or updates were applied that might have caused the issue.
2. Recovery Procedure
1. Check Azure Service Health
Azure Service Health: Navigate to Azure Service Health to check if there are any outages or issues within the Azure region or for the specific resource type you are using. Azure might be experiencing a temporary disruption that could affect your resource.
2. Examine the Resource
- VMs: If it’s a VM, check if it’s not starting, stuck in a certain state, or showing errors like disk corruption.
- Databases: For databases (e.g., Azure SQL Database), check for data inconsistencies, errors related to query failures, or problems with backups/restores.
- Storage Accounts: For storage, verify if data is inaccessible, or if there are issues with blobs, tables, or files.
- Networking: If it’s a networking resource, check for misconfigurations or connectivity issues.
3. Attempt to Resolve the Corruption
For Virtual Machines:
-
Redeploy the VM: This action moves the VM to another host in the Azure datacenter and may resolve issues caused by host-level failures. In the Azure Portal, navigate to the VM > Stop > Redeploy.
-
Restore from Backup/Snapshot: If you have a backup or snapshot, restore the VM to a previous state. If Azure Backup is configured, use it to restore the VM or individual disks.
-
For Azure Storage Accounts: Soft Delete: If the corrupted resource is related to a deleted object (like blobs or containers), you may be able to recover it using the Soft Delete feature. Restore from Backup: If you have Azure Backup configured, restore the storage account or the affected data from backup. Check for Data Consistency: Use tools like AzCopy or Azure Storage Explorer to check data integrity and consistency.
-
For Azure Databases (e.g., SQL Database): Restore from Backup: If using Azure SQL Database, you can restore to a point-in-time from backups. Use the Azure Portal or SQL Management Studio to restore the database from a known good backup. Check Database Integrity: Use SQL commands to check for data corruption or inconsistencies.
See recovery proceedure specific to Database Recovery
4. Recreate the Resource
If restoring the corrupted resource isn’t successful and no backup is available, you may need to delete and recreate the resource.
- Re-deploy VMs or other infrastructure resources.
- Re-create databases, storage, or networking resources as needed.
3. Communication and SLA:
Communicate with End Users: If you are providing services to customers or end-users, you should inform them about the outage, its impact, and the expected resolution time based on Azure's updates.
4. Contact Azure Support
If you cannot resolve the issue or if it’s a critical resource, open a support request with Azure. Provide all relevant information such as error codes, logs, resource IDs, and symptoms to get fast help.
Azure Support can assist with more advanced troubleshooting if the problem is beyond your capabilities.