Tasks for truncating audit-related tables, particularly | (ns metabase.task.truncate-audit-tables
(:require
[clojurewerkz.quartzite.jobs :as jobs]
[clojurewerkz.quartzite.schedule.cron :as cron]
[clojurewerkz.quartzite.triggers :as triggers]
[java-time.api :as t]
[metabase.config :as config]
[metabase.db :as mdb]
[metabase.models.setting :as setting :refer [defsetting]]
[metabase.models.task-history :as task-history]
[metabase.plugins.classloader :as classloader]
[metabase.public-settings.premium-features
:refer [defenterprise]]
[metabase.task :as task]
[metabase.util.i18n :refer [deferred-tru]]
[metabase.util.log :as log]
[toucan2.core :as t2])) |
(set! *warn-on-reflection* true) | |
Load EE implementation if available | (when config/ee-available? (classloader/require 'metabase-enterprise.task.truncate-audit-tables)) |
Minimum allowed value for | (def min-retention-days 30) |
Default value for | (def default-retention-days 720) |
Logs a warning that the value for | (defn log-minimum-value-warning
[env-var-value]
(log/warnf "MB_AUDIT_MAX_RETENTION_DAYS is set to %d; using the minimum value of %d instead."
env-var-value
min-retention-days)) |
(defsetting audit-max-retention-days
(deferred-tru "Number of days to retain data in audit-related tables. Minimum value is 30; set to 0 to retain data indefinitely.")
:visibility :internal
:setter :none
:audit :never
:getter (fn []
(let [env-var-value (setting/get-value-of-type :integer :audit-max-retention-days)]
(cond
(nil? env-var-value)
default-retention-days
;; Treat 0 as an alias for infinity
(zero? env-var-value)
##Inf
(< env-var-value min-retention-days)
(do
(log-minimum-value-warning env-var-value)
min-retention-days)
:else
env-var-value)))
:doc "Sets the maximum number of days Metabase preserves rows for the following application database tables:
- `query_execution`
- `audit_log`
- `view_log`
Twice a day, Metabase will delete rows older than this threshold. The minimum value is 30 days (Metabase will treat entered values of 1 to 29 the same as 30).
If set to 0, Metabase will keep all rows.") | |
(defsetting audit-table-truncation-batch-size (deferred-tru "Batch size to use for deletion of old rows for audit-related tables (like query_execution). Can be only set as an environment variable.") :visibility :internal :setter :none :type :integer :default 50000 :audit :never :export? false) | |
(defn- truncate-table-batched!
[table-name time-column]
(t2/query-one
(case (mdb/db-type)
(:postgres :h2)
{:delete-from (keyword table-name)
:where [:in
:id
{:select [:id]
:from (keyword table-name)
:where [:<=
(keyword time-column)
(t/minus (t/offset-date-time) (t/days (audit-max-retention-days)))]
:limit (audit-table-truncation-batch-size)}]}
(:mysql :mariadb)
{:delete-from (keyword table-name)
:where [:<=
(keyword time-column)
(t/minus (t/offset-date-time) (t/days (audit-max-retention-days)))]
:limit (audit-table-truncation-batch-size)}))) | |
Given a model, deletes all rows older than the configured threshold | (defn- truncate-table!
[model time-column]
(when-not (infinite? (audit-max-retention-days))
(let [table-name (name (t2/table-name model))]
(try
(log/infof "Cleaning up %s table" table-name)
(loop [total-rows-deleted 0]
(let [batch-rows-deleted (truncate-table-batched! table-name time-column)]
;; Only try to delete another batch if the last batch was full
(if (= batch-rows-deleted (audit-table-truncation-batch-size))
(recur (+ total-rows-deleted (long batch-rows-deleted)))
(if (not= total-rows-deleted 0)
(log/infof "%s cleanup successful, %d rows were deleted" table-name total-rows-deleted)
(log/infof "%s cleanup successful, no rows were deleted" table-name)))))
(catch Throwable e
(log/errorf e "%s cleanup failed" table-name)))))) |
List of models to truncate. OSS implementation only truncates | (defenterprise audit-models-to-truncate
metabase-enterprise.task.truncate-audit-tables
[]
[{:model :model/QueryExecution :timestamp-col :started_at}]) |
(defn- truncate-audit-tables!
[]
(run!
(fn [{:keys [model timestamp-col]}]
(task-history/with-task-history {:task "task-history-cleanup"}
(truncate-table! model timestamp-col)))
(audit-models-to-truncate))) | |
Triggers the removal of | (jobs/defjob TruncateAuditTables [_] (truncate-audit-tables!)) |
(def ^:private truncate-audit-tables-job-key "metabase.task.truncate-audit-tables.job") (def ^:private truncate-audit-tables-trigger-key "metabase.task.truncate-audit-tables.trigger") (def ^:private truncate-audit-tables-cron "0 0 */12 * * ? *") ;; Run every 12 hours | |
Run every 12 hours | |
(defmethod task/init! ::TruncateAuditTables [_]
(let [job (jobs/build
(jobs/of-type TruncateAuditTables)
(jobs/with-identity (jobs/key truncate-audit-tables-job-key)))
trigger (triggers/build
(triggers/with-identity (triggers/key truncate-audit-tables-trigger-key))
(triggers/start-now)
(triggers/with-schedule
(cron/schedule
(cron/cron-schedule truncate-audit-tables-cron)
(cron/with-misfire-handling-instruction-do-nothing))))]
(task/schedule-task! job trigger))) | |