FieldValues is used to store a cached list of values of Fields that has There are 2 main classes of FieldValues: Full and Advanced. - Full FieldValues store a list of distinct values of a Field without any constraints. - Whereas Advanced FieldValues has additional constraints: - sandbox: FieldValues of a field but is sandboxed for a specific user - linked-filter: FieldValues for a param that connects to a Field that is constrained by the values of other Field. It's currently being used on Dashboard or Embedding, but it could be used to power any parameters that connect to a Field.
There is also more written about how these are used for remapping in the docstrings for [[metabase.models.params.chain-filter]] and [[metabase.query-processor.middleware.add-dimension-projections]]. | (ns metabase.models.field-values (:require [clojure.string :as str] [java-time.api :as t] [medley.core :as m] [metabase.analyze.core :as analyze] [metabase.db.metadata-queries :as metadata-queries] [metabase.db.query :as mdb.query] [metabase.lib.ident :as lib.ident] [metabase.models.interface :as mi] [metabase.models.serialization :as serdes] [metabase.premium-features.core :refer [defenterprise]] [metabase.query-processor.reducible :as qp.reducible] [metabase.query-processor.schema :as qp.schema] [metabase.util :as u] [metabase.util.date-2 :as u.date] [metabase.util.i18n :refer [tru]] [metabase.util.log :as log] [metabase.util.malli :as mu] [metabase.util.malli.registry :as mr] [metabase.util.malli.schema :as ms] [methodical.core :as methodical] [toucan2.core :as t2])) |
The maximum character length for a stored FieldValues entry. | (def ^:private ^Long entry-max-length 100) |
Maximum total length for a FieldValues entry (combined length of all values for the field). | (def ^:dynamic ^Long *total-max-length* (long (* analyze/auto-list-cardinality-threshold entry-max-length))) |
The absolute maximum number of results to return for a Of course, if a User does something crazy, like mark a million-arity Field as List, we don't want Metabase to explode trying to make their dreams a reality; we need some sort of hard limit to prevent catastrophes. So this limit is effectively a safety to prevent Users from nuking their own instance for Fields that really shouldn't be List Fields at all. For these very-high-cardinality Fields, we're effectively capping the number of FieldValues that get could saved. This number should be a balance of:
| (def ^:dynamic ^Integer *absolute-max-distinct-values-limit* (int 1000)) |
Age of an advanced FieldValues in days.
After this time, these field values should be deleted by the | (def ^java.time.Period advanced-field-values-max-age (t/days 30)) |
How many days until a FieldValues is considered inactive. Inactive FieldValues will not be synced until they are used again. | (def ^:private ^java.time.Period active-field-values-cutoff (t/days 14)) |
A class of fieldvalues that has additional constraints/filters. | (def advanced-field-values-types
#{:sandbox ;; field values filtered by sandbox permissions
:impersonation ;; field values with connection impersonation enforced (db-level roles)
:linked-filter}) ;; field values with constraints from other linked parameters on dashboard/embedding |
field values with constraints from other linked parameters on dashboard/embedding | |
All FieldValues type. | (def ^:private field-values-types
(into #{:full} ;; default type for fieldvalues where it contains values for a field without constraints
advanced-field-values-types)) |
+----------------------------------------------------------------------------------------------------------------+ | Entity & Lifecycle | +----------------------------------------------------------------------------------------------------------------+ | |
(methodical/defmethod t2/table-name :model/FieldValues [_model] :metabase_fieldvalues) | |
(doto :model/FieldValues (derive :metabase/model) (derive :hook/timestamped?)) | |
(t2/deftransforms :model/FieldValues
{:human_readable_values mi/transform-json-no-keywordization
:values mi/transform-json
:type mi/transform-keyword}) | |
(defn- assert-valid-human-readable-values [{human-readable-values :human_readable_values}]
(when-not (mr/validate [:maybe [:sequential [:maybe ms/NonBlankString]]] human-readable-values)
(throw (ex-info (tru "Invalid human-readable-values: values must be a sequence; each item must be nil or a string")
{:human-readable-values human-readable-values
:status-code 400})))) | |
Ensure that type is present, valid, and that a hash_key is provided iff this is an advanced field type. | (defn- assert-valid-type-hash-combo
[{:keys [type hash_key] :as _field-values}]
(when-not (contains? field-values-types type)
(throw (ex-info (tru "Invalid field-values type.")
{:type type
:status-code 400})))
(when (and (= type :full) hash_key)
(throw (ex-info (tru "Full FieldValues shouldn''t have hash_key.")
{:type type
:hash_key hash_key
:status-code 400})))
(when (and (advanced-field-values-types type) (str/blank? hash_key))
(throw (ex-info (tru "Advanced FieldValues require a hash_key.")
{:type type
:status-code 400})))) |
(defn- assert-no-identity-changes [id changes]
(when (some #(contains? changes %) [:field_id :type :hash_key])
(throw (ex-info (tru "Can''t update field_id, type, or hash_key for a FieldValues.")
{:id id
:field_id (:field_id changes)
:type (:type changes)
:hash_key (:hash_key changes)
:status-code 400})))) | |
Remove all advanced FieldValues for a | (defn clear-advanced-field-values-for-field!
[field-or-id]
(t2/delete! :model/FieldValues :field_id (u/the-id field-or-id)
:type [:in advanced-field-values-types])) |
Remove all FieldValues for a | (defn clear-field-values-for-field! [field-or-id] (t2/delete! :model/FieldValues :field_id (u/the-id field-or-id))) |
(t2/define-before-insert :model/FieldValues
[{:keys [field_id] :as field-values}]
(u/prog1 (update field-values :type #(keyword (or % :full)))
(assert-valid-human-readable-values field-values)
(assert-valid-type-hash-combo <>)
;; if inserting a new full fieldvalues, make sure all the advanced field-values of this field are deleted
(when (= :full (:type <>))
(clear-advanced-field-values-for-field! field_id)))) | |
(t2/define-before-update :model/FieldValues
[field-values]
(let [changes (t2/changes field-values)]
(u/prog1 (update field-values :type #(keyword (or % :full)))
(assert-no-identity-changes (:id field-values) changes)
(assert-valid-human-readable-values field-values)
;; if we're updating the values of a Full FieldValues, delete all Advanced FieldValues of this field
(when (and (contains? changes :values) (= :full (:type <>)))
(clear-advanced-field-values-for-field! (:field_id field-values)))))) | |
(defn- assert-coherent-query [{:keys [type hash_key] :as field-values}]
(cond
(nil? type)
(when (some? hash_key)
(throw (ex-info "Invalid query - cannot specify a hash_key without specifying the type"
{:field-values field-values})))
(= :full (keyword type))
(when (some? hash_key)
(throw (ex-info "Invalid query - :full FieldValues cannot have a hash_key"
{:field-values field-values})))
(and (contains? field-values :hash_key) (nil? hash_key))
(throw (ex-info "Invalid query - Advanced FieldValues can only specify a non-empty hash_key"
{:field-values field-values})))) | |
(defn- add-mismatched-hash-filter [{:keys [type] :as field-values}]
(cond
(= :full (keyword type)) (assoc field-values :hash_key nil)
(some? type) (update field-values :hash_key #(or % [:not= nil]))
:else field-values)) | |
(t2/define-before-select :model/FieldValues
[{:keys [kv-args] :as query}]
(assert-coherent-query kv-args)
(update query :kv-args add-mismatched-hash-filter)) | |
(t2/define-after-select :model/FieldValues
[field-values]
(cond-> field-values
(contains? field-values :human_readable_values)
(update :human_readable_values (fn [human-readable-values]
(cond
(sequential? human-readable-values)
human-readable-values
;; in some places human readable values were incorrectly saved as a map. If
;; that's the case, convert them back to a sequence
(map? human-readable-values)
(do
(assert (:values field-values)
(tru ":values must be present to fetch :human_readable_values"))
(mapv human-readable-values (:values field-values)))
;; if the `:human_readable_values` key is present (i.e., if we are fetching the
;; whole row), but `nil`, then replace the `nil` value with an empty vector. The
;; client likes this better.
:else
[]))))) | |
(defmethod serdes/hash-fields :model/FieldValues [_field-values] [(serdes/hydrated-hash :field)]) | |
+----------------------------------------------------------------------------------------------------------------+ | Utils fns | +----------------------------------------------------------------------------------------------------------------+ | |
If FieldValues have not been accessed recently they are considered inactive. | (defn inactive?
[field-values]
(and field-values (t/before? (:last_used_at field-values)
(t/minus (t/offset-date-time) active-field-values-cutoff)))) |
Should this | (defn field-should-have-field-values?
[field-or-field-id]
(if-not (map? field-or-field-id)
(let [field-id (u/the-id field-or-field-id)]
(recur (or (t2/select-one ['Field :base_type :visibility_type :has_field_values] :id field-id)
(throw (ex-info (tru "Field {0} does not exist." field-id)
{:field-id field-id, :status-code 404})))))
(let [{base-type :base_type
visibility-type :visibility_type
has-field-values :has_field_values} field-or-field-id]
(boolean
(and
(not (contains? #{:retired :sensitive :hidden :details-only} (keyword visibility-type)))
(not (isa? (keyword base-type) :type/field-values-unsupported))
(not (= (keyword base-type) :type/*))
(#{:list :auto-list} (keyword has-field-values))))))) |
Like ;; (take-by-length 6 [["Dog"] ["Cat"] ["Duck"]]) ;; => [["Dog"] ["Cat"]] | (defn take-by-length
([max-length]
(fn [rf]
(let [current-length (volatile! 0)]
(fn
([] (rf))
([result]
(rf result))
([result input]
(vswap! current-length + (count (str (first input))))
(if (< @current-length max-length)
(rf result input)
(reduced result)))))))
([max-length coll]
(lazy-seq
(when-let [s (seq coll)]
(let [f (first s)
new-length (- max-length (count (str (first f))))]
(when-not (neg? new-length)
(cons f (take-by-length new-length
(rest s))))))))) |
Field values and human readable values are lists that are zipped together. If the field values have changed, the
human readable values will need to change too. This function reconstructs the | (defn fixup-human-readable-values
[{old-values :values, old-hrv :human_readable_values} new-values]
(when (seq old-hrv)
(let [orig-remappings (zipmap old-values old-hrv)]
(map #(get orig-remappings % (str %)) new-values)))) |
Returns a list of pairs (or single element vectors if there are no humanreadablevalues) for the given
| (defn field-values->pairs
[{:keys [values human_readable_values]}]
(if (seq human_readable_values)
(map vector values human_readable_values)
(map vector values))) |
+----------------------------------------------------------------------------------------------------------------+ | Advanced FieldValues | +----------------------------------------------------------------------------------------------------------------+ | |
Checks if an advanced FieldValues expired. | (defn advanced-field-values-expired?
[fv]
{:pre [(advanced-field-values-types (:type fv))]}
(u.date/older-than? (:created_at fv) advanced-field-values-max-age)) |
Return a hash-key that will be used for sandboxed fieldvalues. | (defenterprise hash-key-for-sandbox metabase-enterprise.sandbox.models.params.field-values [_field-id] nil) |
Return a hash-key that will be used for impersonated fieldvalues. | (defenterprise hash-key-for-impersonation metabase-enterprise.impersonation.driver [_field-id] nil) |
OSS impl of [[hash-key-for-linked-filters]]. | (defn default-hash-key-for-linked-filters
[field-id constraints]
(str (hash [field-id
constraints]))) |
Return a hash-key that will be used for linked-filters fieldvalues. | (defenterprise hash-key-for-linked-filters metabase-enterprise.sandbox.models.params.field-values [field-id constraints] (default-hash-key-for-linked-filters field-id constraints)) |
+----------------------------------------------------------------------------------------------------------------+ | CRUD fns | +----------------------------------------------------------------------------------------------------------------+ | |
(mu/defn- limit-max-char-len-rff :- ::qp.schema/rff
"Returns a rff that will stop when the total character length of the values exceeds `max-char-len`."
[rff max-char-len]
(fn [metadata]
(let [rf (rff metadata)
total-char (volatile! 0)]
(fn
([]
(rf))
([result]
(rf result))
([result row]
(assert (= 1 (count row)))
(vswap! total-char + (count (str (first row))))
(if (> @total-char max-char-len)
(reduced (assoc result ::reached-char-len-limit true))
(rf result row))))))) | |
Fetch a sequence of distinct values for ;; (distinct-values (Field 1)) ;; -> {:values [[1], [2], [3]] :hasmorevalues false} (This function provides the values that normally get saved as a Field's FieldValues. You most likely should not be using this directly in code outside of this namespace, unless it's for a very specific reason, such as certain cases where we fetch ad-hoc FieldValues for GTAP-filtered Fields.) | (defn distinct-values
[field]
(try
(let [result (metadata-queries/table-query (:table_id field)
{:breakout [[:field (u/the-id field) nil]]
:breakout-idents (lib.ident/indexed-idents 1)
:limit *absolute-max-distinct-values-limit*}
(limit-max-char-len-rff qp.reducible/default-rff *total-max-length*))
distinct-values (-> result :data :rows)]
{:values distinct-values
;; has_more_values=true means the list of values we return is a subset of all possible values.
:has_more_values (or (true? (::reached-char-len-limit result))
;; `distinct-values` is from a query
;; with limit = [[*absolute-max-distinct-values-limit*]].
;; So, if the returned `distinct-values` has length equal to that exact limit,
;; we assume the returned values is just a subset of what we have in DB.
(= (count distinct-values)
*absolute-max-distinct-values-limit*))})
(catch Throwable e
(log/error e "Error fetching field values")
nil))) |
Takes a list of field values, return a map of field-id -> latest FieldValues. If a field has more than one Field Values, delete the old ones. This is a workaround for the issue of stale FieldValues rows (metabase#668) In order to mitigate the impact of duplicates, we return the most recently updated row, and delete the older rows. It assumes that all rows are of the same type. Rows could be from multiple field-ids. | (defn- delete-duplicates-and-return-latest!
[fvs]
(let [fvs-grouped-by-field-id (update-vals (group-by :field_id fvs)
#(sort-by :updated_at u/reverse-compare %))
to-delete-fv-ids (->> (vals fvs-grouped-by-field-id)
(mapcat rest)
(map :id))]
(when (seq to-delete-fv-ids)
(t2/delete! :model/FieldValues :id [:in to-delete-fv-ids]))
(update-vals fvs-grouped-by-field-id first))) |
This returns the FieldValues with the given :type and :hash_key for the given Field. This may implicitly delete shadowed entries in the database, see [[delete-duplicates-and-return-latest!]] | (defn- get-latest-field-values
[field-id type hash]
(assert (= (nil? hash) (= type :full)) ":hash_key must be nil iff :type is :full")
(-> (t2/select :model/FieldValues :field_id field-id :type type :hash_key hash)
delete-duplicates-and-return-latest!
(get field-id))) |
This returns the full FieldValues for the given Field. This may implicitly delete shadowed entries in the database, see [[delete-duplicates-and-return-latest!]] | (defn get-latest-full-field-values [field-id] (get-latest-field-values field-id :full nil)) |
Batched version of [[get-latest-full-field-values]] . Takes a list of field-ids and returns a map of field-id -> full FieldValues. This may implicitly delete shadowed entries in the database, see [[delete-duplicates-and-return-latest!]] | (defn batched-get-latest-full-field-values [field-ids] (delete-duplicates-and-return-latest! (t2/select :model/FieldValues :field_id [:in field-ids] :type :full :hash_key nil))) |
Create or update the full FieldValues object for Note that if the full FieldValues are create/updated/deleted, it'll delete all the Advanced FieldValues of the same | (defn create-or-update-full-field-values!
[field & {:keys [field-values human-readable-values]}]
(if (field-should-have-field-values? field)
(let [field-values (or field-values (get-latest-full-field-values (u/the-id field)))
{unwrapped-values :values
:keys [has_more_values]} (distinct-values field)
;; unwrapped-values are 1-tuples, so we need to unwrap their values for storage
values (seq (map first unwrapped-values))
field-name (or (:name field) (:id field))]
(cond
(and values
(= (:values field-values) values)
(= (:has_more_values field-values) has_more_values))
(do
(log/debugf "FieldValues for Field %s remain unchanged. Skipping..." field-name)
::fv-skipped)
;; if the FieldValues object already exists then update values in it
(and field-values values)
(do
(log/debugf "Storing updated FieldValues for Field %s..." field-name)
(t2/update! :model/FieldValues (u/the-id field-values)
(m/remove-vals nil?
{:has_more_values has_more_values
:values values
:human_readable_values (fixup-human-readable-values field-values values)}))
::fv-updated)
;; if FieldValues object doesn't exist create one
values
(do
(log/debugf "Storing FieldValues for Field %s..." field-name)
(mdb.query/select-or-insert! :model/FieldValues {:field_id (u/the-id field), :type :full}
(constantly {:has_more_values has_more_values
:values values
:human_readable_values human-readable-values}))
::fv-created)
;; otherwise this Field isn't eligible, so delete any FieldValues that might exist
:else
(do
(clear-field-values-for-field! field)
::fv-deleted)))
(do
(clear-field-values-for-field! field)
::fv-deleted))) |
Create FieldValues for a | (defn get-or-create-full-field-values!
{:arglists '([field] [field human-readable-values])}
[{field-id :id field-values :values :as field} & [human-readable-values]]
{:pre [(integer? field-id)]}
(when (field-should-have-field-values? field)
(let [existing (or (not-empty field-values) (get-latest-full-field-values field-id))]
(if (or (not existing) (inactive? existing))
(case (create-or-update-full-field-values! field :human-readable-values human-readable-values)
::fv-deleted
nil
::fv-created
(get-latest-full-field-values field-id)
(do
(when existing
(t2/update! :model/FieldValues (:id existing) {:last_used_at :%now}))
(get-latest-full-field-values field-id)))
(do
(t2/update! :model/FieldValues (:id existing) {:last_used_at :%now})
existing))))) |
+----------------------------------------------------------------------------------------------------------------+ | On Demand | +----------------------------------------------------------------------------------------------------------------+ | |
Given a collection of | (defn- table-ids->table-id->is-on-demand?
[table-ids]
(let [table-ids (set table-ids)
table-id->db-id (when (seq table-ids)
(t2/select-pk->fn :db_id 'Table :id [:in table-ids]))
db-id->is-on-demand? (when (seq table-id->db-id)
(t2/select-pk->fn :is_on_demand 'Database
:id [:in (set (vals table-id->db-id))]))]
(into {} (for [table-id table-ids]
[table-id (-> table-id table-id->db-id db-id->is-on-demand?)])))) |
Update the FieldValues for any Fields with | (defn update-field-values-for-on-demand-dbs!
[field-ids]
(let [fields (when (seq field-ids)
(filter field-should-have-field-values?
(t2/select ['Field :name :id :base_type :effective_type :coercion_strategy
:semantic_type :visibility_type :table_id :has_field_values]
:id [:in field-ids])))
table-id->is-on-demand? (table-ids->table-id->is-on-demand? (map :table_id fields))]
(doseq [{table-id :table_id, :as field} fields]
(when (table-id->is-on-demand? table-id)
(log/debugf "Field %s '%s' should have FieldValues and belongs to a Database with On-Demand FieldValues updating."
(u/the-id field) (:name field))
(create-or-update-full-field-values! field))))) |
+----------------------------------------------------------------------------------------------------------------+ | Serialization | +----------------------------------------------------------------------------------------------------------------+ | |
(defmethod serdes/entity-id "FieldValues" [_ _] nil) | |
(defmethod serdes/generate-path "FieldValues" [_ {:keys [field_id]}]
(let [field (t2/select-one 'Field :id field_id)]
(conj (serdes/generate-path "Field" field)
{:model "FieldValues" :id "0"}))) | |
(defmethod serdes/dependencies "FieldValues" [fv] ;; Take the path, but drop the FieldValues section at the end, to get the parent Field's path instead. [(pop (serdes/path fv))]) | |
(defmethod serdes/load-find-local "FieldValues" [path]
;; Delegate to finding the parent Field, then look up its corresponding FieldValues.
(let [field (serdes/load-find-local (pop path))]
;; We only serialize the full values, see [[metabase.models.field/with-values]]
(get-latest-full-field-values (:id field)))) | |
(defn- field-path->field-ref [field-values-path]
(let [[db schema table field :as field-ref] (map :id (pop field-values-path))]
(if field
field-ref
;; It's too short, so no schema. Shift them over and add a nil schema.
[db nil schema table]))) | |
(defmethod serdes/make-spec "FieldValues" [_model-name _opts]
{:copy [:values :human_readable_values :has_more_values :hash_key]
:transform {:created_at (serdes/date)
:last_used_at (serdes/date)
:type (serdes/kw)
:field_id {::serdes/fk true
:export (constantly ::serdes/skip)
:import-with-context (fn [current _ _]
(let [field-ref (field-path->field-ref (serdes/path current))]
(serdes/*import-field-fk* field-ref)))}}}) | |
(defmethod serdes/load-update! "FieldValues" [_ ingested local]
;; It's illegal to change the :type and :hash_key fields, and there's a pre-update check for this.
;; This drops those keys from the incoming FieldValues iff they match the local one. If they are actually different,
;; this preserves the new value so the normal error is produced.
(let [ingested (cond-> ingested
(= (:type ingested) (:type local)) (dissoc :type)
(= (:hash_key ingested) (:hash_key local)) (dissoc :hash_key))]
((get-method serdes/load-update! "") "FieldValues" ingested local))) | |
(def ^:private field-values-slug "___fieldvalues") | |
(defmethod serdes/storage-path "FieldValues" [fv _]
;; [path to table "fields" "field-name___fieldvalues"] since there's zero or one FieldValues per Field, and Fields
;; don't have their own directories.
(let [hierarchy (serdes/path fv)
field-path (serdes/storage-path-prefixes (drop-last hierarchy))]
(update field-path (dec (count field-path)) str field-values-slug))) | |