Automagic DashboardsAutomatically generate questions and dashboards based on predefined heuristics. Note that the primary entry point into this namespace is There are two key inputs to this algorithm:
- An entity to generate the dashboard for. The primary data needed from this entity is:
- The entity type itself
- The field information, especially the metadata about these fields
- This data gathering happens in the Template SelectionWithin the * The template selection is based on the entity type of the item being x-rayed. * Example: X-raying the "ORDERS" table will result in matching the template
This is because the ```clojure (t2/select-one-fn :entity_type :model/Table :name "ORDERS") ;=> :entity/TransactionTable ``` Most tables and all models (as of this writing) will bottom out at ProTip: If you want to introduce a new template type, do the following:
Template FilesTemplate files define a potential dashboard, potentially including titles, seconds, cards, filters, and more.
They are found in
The Dynamic Binding and Dashboard Generation ProcessOnce data has been accreted in Note that if a card template's dimensions, metrics, and filters are not matched to grounded values the card will not be generated. Conversely, if a card template can be matched by multiple combinations of dimensions, multiple cards may be generated. Once a selection of cards have been generated, the top N are selected (default 15), added to the dashboard, and grouped. ExampleThe following example is provided to better illustrate the template process and how dimensions and metrics work. This is a notional dashboard template: Income | __/ Income | # # # X | * |/ | # # # # | * * * +---------- +----------- +----------------- Time Category Y Key things to note: - Each dimension in a card is specified by name. - There are 5 dimensions across all cards: - Income - Time - Category - X - Y - There are 3 metrics: - Count (N Items) - Avg Income - Total Income - Each metric is a computed value based on 0 or more dimensions, also specified by name. - Count is dimensionless - Avg and Total require the Income dimensions - Not shown, but a card such as "Sales by Location" could require 3 dimensions: - Total of the Sales dimension - Longitude and Latitude dimensions - A metric can also have multiple dimensions with its calculated value, such as the quotient of 2 dimensions. - Not described here are filters, which have the same nominal syntax for referencing dimensions as cards and metrics. Dimensions are the key Legoâ„¢ brick for all of the above and are specified as a named element with specialization based on entity and field semantic types as well as a score. For example, Income could have the following potential matches to underlying fields:
- A field from a Sales table with semantic type When matched with actual fields from an x-rayed entity, the highest matching field, by score, is selected to be "bound" to the Income dimensions. Suppose you have an entity of type SalesTable and fields of INCOME (semantic type Income), TAX (type Float), and TOTAL (Float). In this case, the INCOME field would match best (score 100) and be bound to the Income dimension. The other specified dimensions will have similar matching rules. Note that X & Y are, like all other dimensions,
named dimensions. In our above example the Income dimension matched to the INCOME field of type The above example, starting from the dashboard template, works backwards from the actual x-ray generation algorithm but should provide clarity as to the terminology and how everything fits together. In practice, we gather the entity data (including fields), the dashboard templates, attempt to bind dimensions to fields specified in the template, then build metrics, filters, and finally cards based on the bound dimensions. | (ns metabase.xrays.automagic-dashboards.core (:require [clojure.set :as set] [clojure.string :as str] [clojure.walk :as walk] [kixi.stats.core :as stats] [kixi.stats.math :as math] [medley.core :as m] [metabase.analyze.core :as analyze] [metabase.db.query :as mdb.query] [metabase.legacy-mbql.normalize :as mbql.normalize] [metabase.models.field :as field] [metabase.models.interface :as mi] [metabase.query-processor.util :as qp.util] [metabase.util :as u] [metabase.util.i18n :as i18n :refer [tru trun]] [metabase.util.malli :as mu] [metabase.util.malli.schema :as ms] [metabase.xrays.automagic-dashboards.combination :as combination] [metabase.xrays.automagic-dashboards.dashboard-templates :as dashboard-templates] [metabase.xrays.automagic-dashboards.filters :as filters] [metabase.xrays.automagic-dashboards.interesting :as interesting] [metabase.xrays.automagic-dashboards.names :as names] [metabase.xrays.automagic-dashboards.populate :as populate] [metabase.xrays.automagic-dashboards.util :as magic.util] [metabase.xrays.related :as related] [toucan2.core :as t2])) |
(def ^:private public-endpoint "/auto/dashboard/") | |
(def ^:private ^{:arglists '([field])} id-or-name
(some-fn :id :name)) | |
Get user-defined metrics linked to a given entity. | (defmulti linked-metrics
{:arglists '([entity])}
mi/model) |
(defmethod linked-metrics :model/LegacyMetric [{metric-name :name :keys [definition]}]
[{:metric-name metric-name
:metric-title metric-name
:metric-definition definition
:metric-score 100}]) | |
(defmethod linked-metrics :model/Table [{table-id :id}]
(mapcat
linked-metrics
(t2/select :model/LegacyMetric :table_id table-id))) | |
(defmethod linked-metrics :default [_] []) | |
root is a datatype that is an entity augmented with metadata for the purposes of creating an automatic dashboard with respect to that entity. It is called a root because the automated dashboard uses productions to recursively create a tree of dashboard cards to fill the dashboards. This multimethod is for turning a given entity into a root. | (defmulti ->root
{:arglists '([entity])}
mi/model) |
(defmethod ->root :model/Table
[table]
{:entity table
:full-name (:display_name table)
:short-name (:display_name table)
:source table
:database (:db_id table)
:url (format "%stable/%s" public-endpoint (u/the-id table))
:dashboard-templates-prefix ["table"]
:linked-metrics (linked-metrics table)}) | |
(defmethod ->root :model/Segment
[segment]
(let [table (->> segment :table_id (t2/select-one :model/Table :id))]
{:entity segment
:full-name (tru "{0} in the {1} segment" (:display_name table) (:name segment))
:short-name (:display_name table)
:comparison-name (tru "{0} segment" (:name segment))
:source table
:database (:db_id table)
:query-filter [:segment (u/the-id segment)]
:url (format "%ssegment/%s" public-endpoint (u/the-id segment))
:dashboard-templates-prefix ["table"]})) | |
(defmethod ->root :model/LegacyMetric
[metric]
(let [table (->> metric :table_id (t2/select-one :model/Table :id))]
{:entity metric
:full-name (if (:id metric)
(trun "{0} metric" "{0} metrics" (:name metric))
(:name metric))
:short-name (:name metric)
:source table
:database (:db_id table)
;; We use :id here as it might not be a concrete field but rather one from a nested query which
;; does not have an ID.
:url (format "%smetric/%s" public-endpoint (:id metric))
:dashboard-templates-prefix ["metric"]})) | |
(defmethod ->root :model/Field
[field]
(let [table (field/table field)]
{:entity field
:full-name (trun "{0} field" "{0} fields" (:display_name field))
:short-name (:display_name field)
:source table
:database (:db_id table)
;; We use :id here as it might not be a concrete metric but rather one from a nested query
;; which does not have an ID.
:url (format "%sfield/%s" public-endpoint (:id field))
:dashboard-templates-prefix ["field"]})) | |
Is this card or question derived from another model or question? | (def ^:private ^{:arglists '([card-or-question])} nested-query?
(comp some? qp.util/query->source-card-id :dataset_query)) |
Is this card or question native (SQL)? | (def ^:private ^{:arglists '([card-or-question])} native-query?
(comp some? #{:native} qp.util/normalize-token #(get-in % [:dataset_query :type]))) |
(defn- source-question
[card-or-question]
(when-let [source-card-id (qp.util/query->source-card-id (:dataset_query card-or-question))]
(t2/select-one :model/Card :id source-card-id))) | |
(defn- table-like? [card-or-question] (and (nil? (get-in card-or-question [:dataset_query :query :aggregation])) (nil? (get-in card-or-question [:dataset_query :query :breakout])))) | |
Get the Table ID from | (defn- table-id
;; TODO - probably better if we just changed `adhoc-query` to use the same keys as Cards (e.g. `:table_id`) so we
;; didn't need this function, seems like something that would be too easy to forget
[card-or-question]
(or (:table_id card-or-question)
(:table-id card-or-question))) |
(defn- source
[card]
(cond
;; This is a model
(= (:type card) :model) (assoc card :entity_type :entity/GenericTable)
;; This is a query based on a query. Eventually we will want to change this as it suffers from the same sourcing
;; problems as other cards -- The x-ray is not done on the card, but on its source.
(nested-query? card) (-> card
source-question
(assoc :entity_type :entity/GenericTable))
(native-query? card) (-> card (assoc :entity_type :entity/GenericTable))
:else (->> card table-id (t2/select-one :model/Table :id)))) | |
(defmethod ->root :model/Card
[card]
(let [source (source card)]
{:entity card
:source source
:database (:database_id card)
:query-filter (get-in card [:dataset_query :query :filter])
:full-name (tru "\"{0}\"" (:name card))
:short-name (names/source-name {:source source})
:url (format "%s%s/%s" public-endpoint (name (:type source :question)) (u/the-id card))
:dashboard-templates-prefix [(if (table-like? card)
"table"
"question")]})) | |
(defmethod ->root :model/Query
[query]
(let [source (source query)]
{:entity query
:source source
:database (:database-id query)
:query-filter (get-in query [:dataset_query :query :filter])
:full-name (cond
(native-query? query) (tru "Native query")
(table-like? query) (-> source ->root :full-name)
:else (names/question-description {:source source} query))
:short-name (names/source-name {:source source})
:url (format "%sadhoc/%s" public-endpoint
(magic.util/encode-base64-json (:dataset_query query)))
:dashboard-templates-prefix [(if (table-like? query)
"table"
"question")]})) | |
NOTE - This has been lifted to foo. Nuke it here as well. | (defn- fill-templates
[template-type {:keys [root tables]} bindings s]
(let [bindings (some-fn (merge {"this" (-> root
:entity
(assoc :full-name (:full-name root)))}
bindings)
(comp first #(magic.util/filter-tables % tables) dashboard-templates/->entity)
identity)]
(str/replace s #"\[\[(\w+)(?:\.([\w\-]+))?\]\]"
(fn [[_ identifier attribute]]
(let [entity (bindings identifier)
attribute (some-> attribute qp.util/normalize-token)]
(str (or (and (ifn? entity) (entity attribute))
(root attribute)
(interesting/->reference template-type entity)))))))) |
(defn- instantiate-visualization
[[k v] dimensions metrics]
(let [dimension->name (comp vector :name dimensions)
metric->name (comp vector first :metric metrics)]
[k (-> v
(m/update-existing :map.latitude_column dimension->name)
(m/update-existing :map.longitude_column dimension->name)
(m/update-existing :graph.metrics metric->name)
(m/update-existing :graph.dimensions dimension->name))])) | |
Capitalize only the first letter in a given string. | (defn capitalize-first
[s]
(let [s (str s)]
(str (u/upper-case-en (subs s 0 1)) (subs s 1)))) |
(defn- instantiate-metadata
[x context available-metrics bindings]
(-> (walk/postwalk
(fn [form]
(if (i18n/localized-string? form)
(let [s (str form)
new-s (fill-templates :string context bindings s)]
(if (not= new-s s)
(capitalize-first new-s)
s))
form))
x)
(m/update-existing :visualization #(instantiate-visualization % bindings available-metrics)))) | |
Return the set of ids referenced in a cell query | (defn- singular-cell-dimension-field-ids
[{:keys [cell-query]}]
(letfn [(collect-dimensions [[op & args]]
(case (some-> op qp.util/normalize-token)
:and (mapcat collect-dimensions args)
:= (magic.util/collect-field-references args)
nil))]
(->> cell-query
collect-dimensions
(map magic.util/field-reference->id)
set))) |
Return matching dashboard templates ordered by specificity. Most specific is defined as entity type specification the longest ancestor chain. | (defn- matching-dashboard-templates
[dashboard-templates {:keys [source entity]}]
;; Should this be here or lifted to the calling context. It's a magic step.
(let [table-type (or (:entity_type source) :entity/GenericTable)]
(->> dashboard-templates
(filter (fn [{:keys [applies_to]}]
(let [[entity-type field-type] applies_to]
(and (isa? table-type entity-type)
(or (nil? field-type)
(magic.util/field-isa? entity field-type))))))
(sort-by :specificity >)))) |
Return all tables accessible from a given table with the paths to get there. If there are multiple FKs pointing to the same table, multiple entries will be returned. | (defn- linked-tables
[table]
(for [{:keys [id target]} (field/with-targets
(t2/select :model/Field
:table_id (u/the-id table)
:fk_target_field_id [:not= nil]
:active true))
:when (some-> target mi/can-read?)]
(-> target field/table (assoc :link id)))) |
(def ^:private ^{:arglists '([source])} source->db
(comp (partial t2/select-one :model/Database :id) (some-fn :db_id :database_id))) | |
Source fields from tables that are applicable to the entity being x-rayed. | (defn- relevant-fields
[{:keys [source _entity] :as _root} tables]
(let [db (source->db source)]
(if (mi/instance-of? :model/Table source)
(comp (->> (t2/select :model/Field
:table_id [:in (map u/the-id tables)]
:visibility_type "normal"
:preview_display true
:active true)
field/with-targets
(map #(assoc % :db db))
(group-by :table_id))
u/the-id)
(if (table-like? source)
(let [source-fields (->> source
:result_metadata
(map (fn [field]
(as-> field field
(update field :base_type keyword)
(update field :semantic_type keyword)
(mi/instance :model/Field field)
(analyze/run-classifiers field {})
(assoc field :db db)))))]
(constantly source-fields))
(constantly []))))) |
Create the underlying context to which we will add metrics, dimensions, and filters. This is applicable to all dashboard templates. | (defn- make-base-context
[{:keys [source] :as root}]
{:pre [source]}
(let [tables (concat [source] (when (mi/instance-of? :model/Table source)
(linked-tables source)))
table->fields (relevant-fields root tables)]
{:source (assoc source :fields (table->fields source))
:root root
:tables (map #(assoc % :fields (table->fields %)) tables)
:query-filter (filters/inject-refinement (:query-filter root)
(:cell-query root))})) |
(defn- make-dashboard
([root dashboard-template]
(make-dashboard root dashboard-template {:tables [(:source root)] :root root} nil))
([root dashboard-template context {:keys [available-metrics]}]
(-> dashboard-template
(select-keys [:title :description :transient_title :groups])
(cond->
(:comparison? root)
(update :groups (partial m/map-vals (fn [{:keys [title comparison_title] :as group}]
(assoc group :title (or comparison_title title))))))
(instantiate-metadata context available-metrics {})))) | |
Generate a map of satisfiable affinity sets (sets of dimensions that belong together) to visualization types that would be appropriate for each affinity set. | (defn affinities->viz-types
[normalized-card-templates ground-dimensions]
(reduce (partial merge-with set/union)
{}
(for [{:keys [dimensions visualization]} normalized-card-templates
:let [dim-set (into #{} (map ffirst) dimensions)]
:when (every? ground-dimensions dim-set)]
{dim-set #{visualization}}))) |
Create a dashboard group for each user-defined metric. | (defn user-defined-groups
[linked-metrics]
(zipmap (map :metric-name linked-metrics)
(map (fn [{:keys [metric-name]}]
{:title (format "Your %s Metric" metric-name)
:score 0}) linked-metrics))) |
Produce card templates for user-defined metrics. The basic algorithm is to generate the cross product of all user defined metrics to all provided dimension affinities to all potential visualization options for these affinities. | (defn user-defined-metrics->card-templates
[affinities->viz-types user-defined-metrics]
(let [found-summary? (volatile! false)
summary-viz-types #{["scalar" {}] ["smartscalar" {}]}]
(for [[dimension-affinities viz-types] affinities->viz-types
viz viz-types
{:keys [metric-name] :as _user-defined-metric} user-defined-metrics
:let [metric-title (if (seq dimension-affinities)
(format "%s by %s" metric-name
(combination/items->str
(map (fn [s] (format "[[%s]]" s)) (vec dimension-affinities))))
metric-name)
group-name (if (and (not @found-summary?)
(summary-viz-types viz))
(do (vreset! found-summary? true)
"Overview")
metric-name)]]
{:card-score 100
:metrics [metric-name]
:dimensions (mapv (fn [dim] {dim {}}) dimension-affinities)
:visualization viz
:width 6
:title (i18n/->UserLocalizedString metric-title nil {})
:height 4
:group group-name
:card-name (format "Card[%s][%s]" metric-title (first viz))}))) |
Produce the "base" dashboard from the base context for an item and a dashboard template. This includes dashcards and global filters, but does not include related items and is not yet populated. Repeated calls of this might be generated (e.g. the main dashboard and related) then combined once using create dashboard. | (defn generate-base-dashboard
[{{user-defined-metrics :linked-metrics :as root} :root :as base-context}
{template-cards :cards
:keys [dashboard_filters]
:as dashboard-template}
{grounded-dimensions :dimensions
grounded-metrics :metrics
grounded-filters :filters}]
(let [card-templates (interesting/normalize-seq-of-maps :card template-cards)
user-defined-card-templates (user-defined-metrics->card-templates
(affinities->viz-types card-templates grounded-dimensions)
user-defined-metrics)
all-cards (into card-templates user-defined-card-templates)
dashcards (combination/grounded-metrics->dashcards
base-context
all-cards
grounded-dimensions
grounded-filters
grounded-metrics)
template-with-user-groups (update dashboard-template
:groups into (user-defined-groups user-defined-metrics))
empty-dashboard (make-dashboard root template-with-user-groups)]
(assoc empty-dashboard
;; Adds the filters that show at the top of the dashboard
;; Why do we need (or do we) the last remove form?
:filters (->> dashboard_filters
(mapcat (comp :matches grounded-dimensions))
(remove (comp (singular-cell-dimension-field-ids root) id-or-name)))
:cards dashcards))) |
(def ^:private ^:const ^Long max-related 8) (def ^:private ^:const ^Long max-cards 15) (def ^:private ^:const ^Long max-cards-total 30) | |
Turn | (defn ->related-entity
[entity]
(let [{:keys [dashboard-templates-prefix] :as root} (->root entity)
candidate-templates (dashboard-templates/get-dashboard-templates dashboard-templates-prefix)
dashboard-template (->> root
(matching-dashboard-templates candidate-templates)
first)
dashboard (make-dashboard root dashboard-template)]
{:url (:url root)
:title (:full-name root)
:description (:description dashboard)})) |
(defn- related-entities
[root]
(-> root
:entity
related/related
(update :fields (partial remove magic.util/key-col?))
(->> (m/map-vals (comp (partial map ->related-entity) u/one-or-many))))) | |
(mu/defn- indepth
[{:keys [dashboard-templates-prefix url] :as root}
{:keys [dashboard-template-name]} :- [:maybe dashboard-templates/DashboardTemplate]]
(let [base-context (make-base-context root)]
(->> (dashboard-templates/get-dashboard-templates (concat dashboard-templates-prefix [dashboard-template-name]))
(keep (fn [{indepth-template-name :dashboard-template-name
template-dimensions :dimensions
template-metrics :metrics
template-filters :filters
:as indepth}]
(let [grounded-values (interesting/identify
base-context
{:dimension-specs template-dimensions
:metric-specs template-metrics
:filter-specs template-filters})
{:keys [description cards] :as dashboard} (generate-base-dashboard
base-context
indepth
grounded-values)]
(when (and description (seq cards))
{:title ((some-fn :short-title :title) dashboard)
:description description
:url (format "%s/rule/%s/%s" url dashboard-template-name indepth-template-name)}))))
(hash-map :indepth)))) | |
(defn- drilldown-fields
[root available-dimensions]
(when (->> root :source (mi/instance-of? :model/Table))
(->> available-dimensions
vals
(mapcat :matches)
(filter mi/can-read?)
filters/interesting-fields
(map ->related-entity)
(hash-map :drilldown-fields)))) | |
(defn- comparisons
[root]
{:compare (concat
(for [segment (->> root :entity related/related :segments (map ->root))]
{:url (str (:url root) "/compare/segment/" (-> segment :entity u/the-id))
:title (tru "Compare with {0}" (:comparison-name segment))
:description })
(when ((some-fn :query-filter :cell-query) root)
[{:url (if (->> root :source (mi/instance-of? :model/Table))
(str (:url root) "/compare/table/" (-> root :source u/the-id))
(str (:url root) "/compare/adhoc/"
(magic.util/encode-base64-json
{:database (:database root)
:type :query
:query {:source-table (->> root
:source
u/the-id
(str "card__"))}})))
:title (tru "Compare with entire dataset")
:description }]))}) | |
We fill available slots round-robin style. Each selector is a list of fns that are tried against
| (defn- fill-related
[available-slots selectors related]
(let [pop-first (fn [m ks]
(loop [[k & ks] ks]
(let [item (-> k m first)]
(cond
item [item (update m k rest)]
(empty? ks) [nil m]
:else (recur ks)))))
count-leafs (comp count (partial mapcat val))
[selected related] (reduce-kv
(fn [[selected related] k v]
(loop [[selector & remaining-selectors] v
related related
selected selected]
(let [[next related] (pop-first related (mapcat shuffle selector))
num-selected (count-leafs selected)]
(cond
(= num-selected available-slots)
(reduced [selected related])
next
(recur remaining-selectors related (update selected k conj next))
(empty? remaining-selectors)
[selected related]
:else
(recur remaining-selectors related selected)))))
[{} related]
selectors)
num-selected (count-leafs selected)]
(if (pos? num-selected)
(merge-with concat
selected
(fill-related (- available-slots num-selected) selectors related))
{}))) |
(def ^:private related-selectors
{:model/Table (let [down [[:indepth] [:segments :metrics] [:drilldown-fields]]
sideways [[:linking-to :linked-from] [:tables]]
compare [[:compare]]]
{:zoom-in [down down down down]
:related [sideways sideways]
:compare [compare compare]})
:model/Segment (let [down [[:indepth] [:segments :metrics] [:drilldown-fields]]
sideways [[:linking-to] [:tables]]
up [[:table]]
compare [[:compare]]]
{:zoom-in [down down down]
:zoom-out [up]
:related [sideways sideways]
:compare [compare compare]})
:model/LegacyMetric (let [down [[:drilldown-fields]]
sideways [[:metrics :segments]]
up [[:table]]
compare [[:compare]]]
{:zoom-in [down down]
:zoom-out [up]
:related [sideways sideways sideways]
:compare [compare compare]})
:model/Field (let [sideways [[:fields]]
up [[:table] [:metrics :segments]]
compare [[:compare]]]
{:zoom-out [up]
:related [sideways sideways]
:compare [compare]})
:model/Card (let [down [[:drilldown-fields]]
sideways [[:metrics] [:similar-questions :dashboard-mates]]
up [[:table]]
compare [[:compare]]]
{:zoom-in [down down]
:zoom-out [up]
:related [sideways sideways sideways]
:compare [compare compare]})
:model/Query (let [down [[:drilldown-fields]]
sideways [[:metrics] [:similar-questions]]
up [[:table]]
compare [[:compare]]]
{:zoom-in [down down]
:zoom-out [up]
:related [sideways sideways sideways]
:compare [compare compare]})}) | |
Build a balanced list of related X-rays. General composition of the list is determined for each
root type individually via | (mu/defn- related
[root
available-dimensions
dashboard-template :- [:maybe dashboard-templates/DashboardTemplate]]
(->> (merge (indepth root dashboard-template)
(drilldown-fields root available-dimensions)
(related-entities root)
(comparisons root))
(fill-related max-related (get related-selectors (-> root :entity mi/model))))) |
Return a map of fields referenced in filter clause. | (defn- filter-referenced-fields
[root filter-clause]
(->> filter-clause
magic.util/collect-field-references
(map (fn [[_ id-or-name _options]]
[id-or-name (magic.util/->field root id-or-name)]))
(remove (comp nil? second))
(into {}))) |
Produce a fully-populated dashboard from the base context for an item and a dashboard template. | (defn generate-dashboard
[{{:keys [show url query-filter] :as root} :root :as base-context}
{:as dashboard-template}
{grounded-dimensions :dimensions :as grounded-values}]
(let [show (or show max-cards)
dashboard (generate-base-dashboard base-context dashboard-template grounded-values)]
(-> dashboard
(populate/create-dashboard show)
(assoc
:related (related
root grounded-dimensions
dashboard-template)
:more (when (and (not= show :all)
(-> dashboard :cards count (> show)))
(format "%s#show=all" url))
:transient_filters query-filter
:param_fields (filter-referenced-fields root query-filter)
:auto_apply_filters true
:width "fixed")))) |
Create dashboards for table | (defn- automagic-dashboard
[{:keys [dashboard-template dashboard-templates-prefix] :as root}]
(let [base-context (make-base-context root)
{template-dimensions :dimensions
template-metrics :metrics
template-filters :filters
:as template} (if dashboard-template
(dashboard-templates/get-dashboard-template dashboard-template)
(first (matching-dashboard-templates
(dashboard-templates/get-dashboard-templates dashboard-templates-prefix)
root)))
grounded-values (interesting/identify
base-context
{:dimension-specs template-dimensions
:metric-specs template-metrics
:filter-specs template-filters})]
(generate-dashboard base-context template grounded-values))) |
Create a transient dashboard analyzing given entity. This function eventually calls out to | (defmulti automagic-analysis
{:arglists '([entity opts])}
(fn [entity _]
(mi/model entity))) |
(defmethod automagic-analysis :model/Table [table opts] (automagic-dashboard (merge (->root table) opts))) | |
(defmethod automagic-analysis :model/Segment [segment opts] (automagic-dashboard (merge (->root segment) opts))) | |
(defmethod automagic-analysis :model/LegacyMetric [metric opts] (automagic-dashboard (merge (->root metric) opts))) | |
(mu/defn- collect-metrics :- [:maybe [:sequential (ms/InstanceOf :model/LegacyMetric)]]
[root question]
(map (fn [aggregation-clause]
(if (-> aggregation-clause
first
qp.util/normalize-token
(= :metric))
(->> aggregation-clause second (t2/select-one :model/LegacyMetric :id))
(let [table-id (table-id question)]
(mi/instance :model/LegacyMetric {:definition {:aggregation [aggregation-clause]
:source-table table-id}
:name (names/metric->description root aggregation-clause)
:table_id table-id}))))
(get-in question [:dataset_query :query :aggregation]))) | |
(mu/defn- collect-breakout-fields :- [:maybe [:sequential (ms/InstanceOf :model/Field)]]
[root question]
(for [breakout (get-in question [:dataset_query :query :breakout])
field-clause (take 1 (magic.util/collect-field-references breakout))
:let [field (magic.util/->field root field-clause)]
:when (and field
(= (:table_id field) (table-id question)))]
field)) | |
(defn- decompose-question
[root question opts]
(letfn [(analyze [x]
(try
(automagic-analysis x (assoc opts
:source (:source root)
:query-filter (:query-filter root)
:database (:database root)))
(catch Throwable e
(throw (ex-info (tru "Error decomposing question: {0}" (ex-message e))
{:root root, :question question, :object x}
e)))))]
(into []
(comp cat (map analyze))
[(collect-metrics root question)
(collect-breakout-fields root question)]))) | |
Ensure that elements of an original dataset query are preserved in dashcard queries. | (defn- preserve-entity-element
[dashboard entity entity-element]
(if-let [element-value (get-in entity [:dataset_query :query entity-element])]
(letfn [(splice-element [dashcard]
(cond-> dashcard
(get-in dashcard [:card :dataset_query :query])
(update-in [:card :dataset_query :query entity-element]
(fnil into (empty element-value))
element-value)))]
(update dashboard :dashcards (partial map splice-element)))
dashboard)) |
(defn- query-based-analysis
[{:keys [entity] :as root} opts {:keys [cell-query cell-url]}]
(let [transient-dash (if (table-like? entity)
(let [root' (merge root
(when cell-query
{:url cell-url
:entity (:source root)
:dashboard-templates-prefix ["table"]})
opts)]
(automagic-dashboard root'))
(let [opts (assoc opts :show max-cards-total)
root' (merge root
(when cell-query
{:url cell-url})
opts)
base-dash (automagic-dashboard root')
dash (reduce populate/merge-dashboards
base-dash
(decompose-question root entity opts))]
(merge dash
(when cell-query
(let [title (tru "A closer look at {0}" (names/cell-title root cell-query))]
{:transient_name title
:name title})))))]
(-> transient-dash
(preserve-entity-element (:entity root) :joins)
(preserve-entity-element (:entity root) :expressions)))) | |
(defmethod automagic-analysis :model/Card
[card {:keys [cell-query] :as opts}]
(let [root (->root card)
cell-url (format "%squestion/%s/cell/%s" public-endpoint
(u/the-id card)
(magic.util/encode-base64-json cell-query))]
(query-based-analysis root opts
(when cell-query
{:cell-query cell-query
:cell-url cell-url})))) | |
(defmethod automagic-analysis :model/Query
[query {:keys [cell-query] :as opts}]
(let [root (->root query)
cell-query (when cell-query (mbql.normalize/normalize-fragment [:query :filter] cell-query))
opts (cond-> opts
cell-query (assoc :cell-query cell-query))
cell-url (format "%sadhoc/%s/cell/%s" public-endpoint
(magic.util/encode-base64-json (:dataset_query query))
(magic.util/encode-base64-json cell-query))]
(query-based-analysis root opts
(when cell-query
{:cell-query cell-query
:cell-url cell-url})))) | |
(defmethod automagic-analysis :model/Field [field opts] (automagic-dashboard (merge (->root field) opts))) | |
Add a stats field to each provided table with the following data: - num-fields: The number of Fields in each table - list-like?: Is this field 'list like' - link-table?: Is every Field a foreign key to another table | (defn- enhance-table-stats
[tables]
(when (not-empty tables)
(let [field-count (->> (mdb.query/query {:select [:table_id [:%count.* "count"]]
:from [:metabase_field]
:where [:and [:in :table_id (map u/the-id tables)]
[:= :active true]]
:group-by [:table_id]})
(into {} (map (juxt :table_id :count))))
list-like? (->> (when-let [candidates (->> field-count
(filter (comp (partial >= 2) val))
(map key)
not-empty)]
(mdb.query/query {:select [:table_id]
:from [:metabase_field]
:where [:and [:in :table_id candidates]
[:= :active true]
[:or [:not= :semantic_type "type/PK"]
[:= :semantic_type nil]]]
:group-by [:table_id]
:having [:= :%count.* 1]}))
(into #{} (map :table_id)))
;; Table comprised entierly of join keys
link-table? (when (seq field-count)
(->> (mdb.query/query {:select [:table_id [:%count.* "count"]]
:from [:metabase_field]
:where [:and [:in :table_id (keys field-count)]
[:= :active true]
[:in :semantic_type ["type/PK" "type/FK"]]]
:group-by [:table_id]})
(filter (fn [{:keys [table_id count]}]
(= count (field-count table_id))))
(into #{} (map :table_id))))]
(for [table tables]
(let [table-id (u/the-id table)]
(assoc table :stats {:num-fields (field-count table-id 0)
:list-like? (boolean (contains? list-like? table-id))
:link-table? (boolean (contains? link-table? table-id))})))))) |
Maximal number of tables per schema shown in | (def ^:private ^:const ^Long max-candidate-tables 10) |
Return a list of tables in database with ID Tables are ranked based on how specific dashboard template has been used, and the number of fields. Schemes are ranked based on the number of distinct entity types and the interestingness of tables they contain (see above). | (defn candidate-tables
([database] (candidate-tables database nil))
([database schema]
(let [dashboard-templates (dashboard-templates/get-dashboard-templates ["table"])]
(->> (apply t2/select [:model/Table :id :schema :display_name :entity_type :db_id]
(cond-> [:db_id (u/the-id database)
:visibility_type nil
:active true]
schema (concat [:schema schema])))
(filter mi/can-read?)
enhance-table-stats
(remove (comp (some-fn :link-table? (comp zero? :num-fields)) :stats))
(map (fn [table]
(let [root (->root table)
{:keys [dashboard-template-name]
:as dashboard-template} (->> root
(matching-dashboard-templates dashboard-templates)
first)
dashboard (make-dashboard root dashboard-template)]
{:url (format "%stable/%s" public-endpoint (u/the-id table))
:title (:short-name root)
:score (+ (math/sq (:specificity dashboard-template))
(math/log (-> table :stats :num-fields))
(if (-> table :stats :list-like?)
-10
0))
:description (:description dashboard)
:table table
:dashboard-template-name dashboard-template-name})))
(group-by (comp :schema :table))
(map (fn [[schema tables]]
(let [tables (->> tables
(sort-by :score >)
(take max-candidate-tables))]
{:id (format "%s/%s" (u/the-id database) schema)
:tables tables
:schema schema
:score (+ (math/sq (transduce (m/distinct-by :dashboard-template-name)
stats/count
tables))
(math/sqrt (transduce (map (comp math/sq :score))
stats/mean
tables)))})))
(sort-by :score >))))) |