From 0a720fa90f5f1682d59a7840d5541e40d0a79500 Mon Sep 17 00:00:00 2001 From: Alex Barreto Date: Fri, 22 Oct 2021 14:27:20 -0500 Subject: [PATCH] DBFS defaults edit for Cluster Logs best practices --- toc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/toc.md b/toc.md index 8c830da..02b2e4e 100644 --- a/toc.md +++ b/toc.md @@ -412,7 +412,7 @@ You should treat Init scripts with *extreme* caution because they can easily lea ## Use Cluster Log Delivery Feature to Manage Logs *Impact: Medium* -By default, Cluster logs are sent to default DBFS but you should consider sending the logs to a blob store location under your control using the [Cluster Log Delivery](https://docs.azuredatabricks.net/user-guide/clusters/log-delivery.html#cluster-log-delivery) feature. The Cluster Logs contain logs emitted by user code, as well as Spark framework’s Driver and Executor logs. Sending them to a blob store controlled by yourself is recommended over default DBFS location because: +By default, Cluster logs are sent to DBFS by default (```dbfs:/cluster-logs```) but you should consider sending the logs to a blob store location under your control using the [Cluster Log Delivery](https://docs.azuredatabricks.net/user-guide/clusters/log-delivery.html#cluster-log-delivery) feature. The Cluster Logs contain logs emitted by user code, as well as Spark framework’s Driver and Executor logs. Sending them to a blob store controlled by yourself is recommended over default DBFS location because: 1. ADB’s automatic 30-day default DBFS log purging policy might be too short for certain compliance scenarios. A blob store loction in your subscription will be free from such policies. 2. You can ship logs to other tools only if they are present in your storage account and a resource group governed by you. The root DBFS, although present in your subscription, is launched inside a Microsoft Azure managed resource group and is protected by a read lock. Because of this lock the logs are only accessible by privileged Azure Databricks framework code. However, constructing a pipeline to ship the logs to downstream log analytics tools requires logs to be in a lock-free location first.