diff --git a/docs/indexes/assets/dynamic-index-fields-1.png b/docs/indexes/assets/dynamic-index-fields-1.png index b13d9076f8..a20adda0ec 100644 Binary files a/docs/indexes/assets/dynamic-index-fields-1.png and b/docs/indexes/assets/dynamic-index-fields-1.png differ diff --git a/docs/indexes/assets/dynamic-index-fields-2.png b/docs/indexes/assets/dynamic-index-fields-2.png index dd2b4082ea..c5b1bc14bc 100644 Binary files a/docs/indexes/assets/dynamic-index-fields-2.png and b/docs/indexes/assets/dynamic-index-fields-2.png differ diff --git a/docs/indexes/content/_using-analyzers-csharp.mdx b/docs/indexes/content/_using-analyzers-csharp.mdx index 6022cfd145..2e824fea4b 100644 --- a/docs/indexes/content/_using-analyzers-csharp.mdx +++ b/docs/indexes/content/_using-analyzers-csharp.mdx @@ -24,7 +24,7 @@ import CodeBlock from '@theme/CodeBlock'; * [Understanding the role of analyzers](../../indexes/using-analyzers.mdx#understanding-the-role-of-analyzers) * [Analyzers available in RavenDB](../../indexes/using-analyzers.mdx#analyzers-available-in-ravendb) * [Setting analyzer for index-field](../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field) - * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendb) + * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendbs-default-analyzers) * [Disabling indexing for index-field](../../indexes/using-analyzers.mdx#disabling-indexing-for-index-field) * [Creating custom analyzers](../../indexes/using-analyzers.mdx#creating-custom-analyzers) * [Viewing the indexed terms](../../indexes/using-analyzers.mdx#viewing-the-indexed-terms) diff --git a/docs/indexes/content/_using-analyzers-nodejs.mdx b/docs/indexes/content/_using-analyzers-nodejs.mdx index 6afd3b6202..20ac320455 100644 --- a/docs/indexes/content/_using-analyzers-nodejs.mdx +++ b/docs/indexes/content/_using-analyzers-nodejs.mdx @@ -24,7 +24,7 @@ import CodeBlock from '@theme/CodeBlock'; * [Understanding the role of analyzers](../../indexes/using-analyzers.mdx#understanding-the-role-of-analyzers) * [Analyzers available in RavenDB](../../indexes/using-analyzers.mdx#analyzers-available-in-ravendb) * [Setting analyzer for index-field](../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field) - * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendb) + * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendbs-default-analyzers) * [Disabling indexing for index-field](../../indexes/using-analyzers.mdx#disabling-indexing-for-index-field) * [Creating custom analyzers](../../indexes/using-analyzers.mdx#creating-custom-analyzers) * [Viewing the indexed terms](../../indexes/using-analyzers.mdx#viewing-the-indexed-terms) diff --git a/docs/indexes/content/_using-dynamic-fields-csharp.mdx b/docs/indexes/content/_using-dynamic-fields-csharp.mdx index b4c4f518df..5bf052d482 100644 --- a/docs/indexes/content/_using-dynamic-fields-csharp.mdx +++ b/docs/indexes/content/_using-dynamic-fields-csharp.mdx @@ -2,533 +2,923 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. - -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. - -* In this page: - - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -## Indexing documents fields KEYS - -## Example - index any field under object - - + -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + + +### Example - index every field under an object + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. * **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - - // The KEYS under the Attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - public Dictionary Attributes \{ get; set; \} -\} -`} - - - - -{`// Sample document content -\{ - "Attributes": \{ - "Color": "Red", - "Size": 42 - \} -\} -`} - - + + + ```csharp + public class Product + { + public string Id { get; set; } + + // The KEYS under the Attributes object will be dynamically indexed + // Fields added to this object after index creation time will also get indexed + public Dictionary Attributes { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "Attributes": { + "Color": "Red", + "Size": 42 + } + } + ``` + * **The index**: - The below index will index any field under the `Attributes` object from the document, + + The following index will index any field under the `Attributes` object from the document, a dynamic-index-field will be created for each such field. New fields added to the object after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. - - - -{`public class Products_ByAttributeKey : AbstractIndexCreationTask -{ - public Products_ByAttributeKey() + + + ```csharp + public class Products_ByAttributeKey : AbstractIndexCreationTask { - Map = products => from p in products - select new - { - // Call 'CreateField' to generate dynamic-index-fields from the Attributes object keys - // Using '_' is just a convention. Any other string can be used instead of '_' - - // The actual field name will be item.Key - // The actual field terms will be derived from item.Value - _ = p.Attributes.Select(item => CreateField(item.Key, item.Value)) - }; + public Products_ByAttributeKey() + { + Map = products => from p in products + select new + { + // Call 'CreateField' to generate dynamic-index-fields + // from the Attributes object keys. + + // Using '_' is just a convention. + // Any other string can be used instead of '_' + + // The actual field name will be 'item.Key' + // The actual field terms will be derived from 'item.Value' + _ = p.Attributes.Select(item => CreateField(item.Key, item.Value)) + }; + } } -} -`} - - - - -{`public class Products_ByAttributeKey_JS : AbstractJavaScriptIndexCreationTask -{ - public Products_ByAttributeKey_JS() + ``` + + + ```csharp + public class Products_ByAttributeKey_JS : AbstractJavaScriptIndexCreationTask { - Maps = new HashSet + public Products_ByAttributeKey_JS() { - @"map('Products', function (p) { - return { - _: Object.keys(p.Attributes).map(key => createField(key, p.Attributes[key], - { indexing: 'Search', storage: true, termVector: null })) - }; - })" - }; + Maps = new HashSet + { + @"map('Products', function (p) { + return { + // The field name will be the key + // The field terms will be derived from the corresponding value + _: Object.keys(p.Attributes) + .map(key => createField(key, p.Attributes[key])) + }; + })" + }; + } } -} -`} - - - - + ``` + + * **The query**: - * You can now query the generated dynamic-index fields. - The `_` property is Not queryable but used only in the index definition syntax. - * To get all documents with some 'Size' use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'Size' is a dynamic-index-field that was indexed from the Attributes object - .WhereEquals("Size", 42) - .ToList(); -`} - - - - -{`// 'Size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey' where Size = 42 -`} - - - + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _Size_ use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'Size' is a dynamic-index-field that was indexed from the Attributes object + .WhereEquals("Size", 42) + .ToList(); + ``` + + + ```sql + // 'Size' is a dynamic-index-field that was indexed from the Attributes object + from index 'Products/ByAttributeKey' where Size = 42 + ``` + + + + + Dynamic index fields are queried with _DocumentQuery_ or _RQL_ because the field names (e.g. `Size`, `Color`) + are generated at indexing time and are not properties on the _Product_ class. + + A strongly typed LINQ query such as `Query().Where(p => p.Size == 42)` would not compile. + Use the string-based `WhereEquals("Size", 42)` or RQL `where Size = 42` instead. + -## Example - index any field + - + + +### Example - index every field + +The following example allows you to: * Define an index on a collection **without** needing any common structure between the indexed documents. - * After index is deployed, any new field added to the document will be indexed as well. - + * After the index is deployed, any new field added to any document in the collection will be indexed as well. - + -Consider whether this is really necessary, as indexing every single field can end up costing time and disk space. +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - public string FirstName \{ get; set; \} - public string LastName \{ get; set; \} - public string Title \{ get; set; \} - // ... -\} -`} - - - - - -{`// Sample document content - \{ - "FirstName": "John", - "LastName": "Doe", - "Title": "Engineer", + + + ```csharp + public class Product + { + public string Id { get; set; } + + // All KEYS in the document will be dynamically indexed + // Fields added to the document after index creation time will also get indexed + public string FirstName { get; set; } + public string LastName { get; set; } + public string Title { get; set; } // ... -\} -`} - - + } + ``` + + + + ```json + // Sample document content + { + "FirstName": "John", + "LastName": "Doe", + "Title": "Engineer", + // ... + } + ``` + * **The index**: - The below index will index any field from the document, + + The following index will index any field from the document, a dynamic-index-field will be created for each field. New fields added to the document after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. + + The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection. - - - -{`public class Products_ByAnyField_JS : AbstractJavaScriptIndexCreationTask -{ - public Products_ByAnyField_JS() + + + ```csharp + public class Products_ByAnyField_JS : AbstractJavaScriptIndexCreationTask { - // This will index EVERY FIELD under the top level of the document - Maps = new HashSet + public Products_ByAnyField_JS() { - @"map('Products', function (p) { - return { - _: Object.keys(p).map(key => createField(key, p[key], - { indexing: 'Search', storage: true, termVector: null })) - } - })" - }; + // This will index EVERY FIELD under the top level of the document + Maps = new HashSet + { + @"map('Products', function (p) { + return { + _: Object.keys(p).map(key => createField(key, p[key])) + }; + })" + }; + } } -} -`} - - - + ``` + + * **The query**: - * To get all documents with some 'LastName' use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'LastName' is a dynamic-index-field that was indexed from the document - .WhereEquals("LastName", "Doe") - .ToList(); -`} - - - - -{`// 'LastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where LastName = "Doe" -`} - - - + + To get all documents with a specific _LastName_ use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'LastName' is a dynamic-index-field that was indexed from the document + .WhereEquals("LastName", "Doe") + .ToList(); + ``` + + + ```sql + // 'LastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where LastName = "Doe" + ``` + + + + + + + + + +### Example - basic -## Indexing documents fields VALUES + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + -## Example - basic +* **The document**: + + + ```csharp + public class Product + { + public string Id { get; set; } + + // The VALUE of ProductType will be dynamically indexed + public string ProductType { get; set; } + public int PricePerUnit { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "ProductType": "Electronics", + "PricePerUnit": 23 + } + ``` + - +* **The index**: + + The following index will index the **value** of document field 'ProductType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + + ```csharp + public class Products_ByProductType : AbstractIndexCreationTask + { + public Products_ByProductType() + { + Map = products => from p in products + select new + { + // Call 'CreateField' to generate the dynamic-index-fields + // The field name will be the value of document field 'ProductType' + // The field terms will be derived from document field 'PricePerUnit' + _ = CreateField(p.ProductType, p.PricePerUnit) + }; + } + } + ``` + + + ```csharp + public class Products_ByProductType_JS : AbstractJavaScriptIndexCreationTask + { + public Products_ByProductType_JS() + { + Maps = new HashSet + { + @"map('Products', function (p) { + return { + // The field name will be the value of document field 'ProductType' + // The field terms will be derived from document field 'PricePerUnit' + _: createField(p.ProductType, p.PricePerUnit) + }; + })" + }; + } + } + ``` + + + +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'ProductType' + .WhereEquals("Electronics", 23) + .ToList(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'ProductType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. -* For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---index-a-list-of-properties) below. +### Example - list + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. * **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - - // The VALUE of ProductType will be dynamically indexed - public string ProductType \{ get; set; \} - public int PricePerUnit \{ get; set; \} -\} -`} - - - - - -{`// Sample document content -\{ - "ProductType": "Electronics", - "PricePerUnit": 23 -\} -`} - - + + + ```csharp + public class Product + { + public string Id { get; set; } + public string Name { get; set; } + + // For each element in this list, + // the VALUE of property 'PropName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + public List Attributes { get; set; } + } + + public class Attribute + { + public string PropName { get; set; } + public string PropValue { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "Name": "SomeName", + "Attributes": [ + { + "PropName": "Color", + "PropValue": "Blue" + }, + { + "PropName": "Width", + "PropValue": "10" + }, + { + "PropName": "Length", + "PropValue": "20" + }, + ... + ] + } + ``` + * **The index**: - The below index will index the **value** of document field 'ProductType'. + + The following index will create a dynamic-index-field per item in the document's `Attributes` list. + New items added to the Attributes list after index creation time will be dynamically indexed as well. - This value will be the dynamic-index-field name on which you can query. - E.g., Field value `Electronics` will be the dynamic-index-field. + The actual dynamic-index-field name on which you can query will be the item's PropName **value**. + E.g., 'PropName' value `Width` will be a dynamic-index-field. + + + + ```csharp + public class Attributes_ByName : AbstractIndexCreationTask + { + public Attributes_ByName() + { + Map = products => from a in products + select new + { + // Define the dynamic-index-fields by calling 'CreateField' + // A dynamic-index-field will be generated for each item in 'Attributes' + + // For each item, the field name will be the value of field 'PropName' + // The field terms will be derived from field 'PropValue' + _ = a.Attributes.Select(item => + CreateField(item.PropName, item.PropValue)), + + // A regular index field can be defined as well: + Name = a.Name + }; + } + } + ``` + + + ```csharp + public class Attributes_ByName_JS : AbstractJavaScriptIndexCreationTask + { + public Attributes_ByName_JS() + { + Maps = new HashSet + { + @"map('Products', function (p) { + return { + // For each item, + // the field name will be the value of field 'PropName' + // The field terms will be derived from field 'PropValue' + _: p.Attributes.map(item => createField( + item.PropName, item.PropValue)), + + // A regular index field can be defined as well: + Name: p.Name + }; + })" + }; + } + } + ``` + + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'Width' is a dynamic-index-field that was indexed from the Attributes list + .WhereEquals("Width", 10) + .ToList(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the Attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `FieldIndexing.Search` in `CreateFieldOptions`. + +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `AllFields` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `AllFields` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. + + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. + + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + + + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```csharp + public class Product + { + public string Id { get; set; } + + public Dictionary Descriptions { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "Descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `Descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. + + Each generated dynamic field is indexed for **full-text search**. + + + + ```csharp + public class Products_ByLocalizedDescription : AbstractIndexCreationTask + { + private const string UseSearchAnalyzerForDynamicFields = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription() + { + Map = products => from product in products + select new + { + _ = product.Descriptions.Select(x => + CreateField( + "Description_" + x.Key, + x.Value, + new CreateFieldOptions + { + // Index each generated dynamic field for FTS + Indexing = FieldIndexing.Search, + Storage = FieldStorage.No + })) + }; + + StoreAllFields(FieldStorage.No); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + Configuration[UseSearchAnalyzerForDynamicFields] = "true"; + } + } + ``` + + + ```csharp + public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask + { + private const string UseSearchAnalyzerForDynamicFields = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription_JS() + { + Maps = new HashSet + { + @"map('Products', function (product) { + return { + _: Object.keys(product.Descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: 'Search', + storage: 'No' + }); + }) + }; + })" + }; + + Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions + { + Storage = FieldStorage.No + }; + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + Configuration[UseSearchAnalyzerForDynamicFields] = "true"; + } + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```csharp + List results = session.Advanced + .DocumentQuery() + .Search("Description_English", "north wind") + .ToList(); + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + + +--- + + + +### Configuring analyzers for dynamic fields - - - -{`public class Products_ByProductType : AbstractIndexCreationTask +`CreateFieldOptions` does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `AllFields` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `AllFields` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field + + + +```csharp +public class Products_ByLocalizedDescription : AbstractIndexCreationTask { - public Products_ByProductType() + public Products_ByLocalizedDescription() { - Map = products => from p in products + Map = products => from product in products select new { - // Call 'CreateField' to generate the dynamic-index-fields - // The field name will be the value of document field 'ProductType' - // The field terms will be derived from document field 'PricePerUnit' - _ = CreateField(p.ProductType, p.PricePerUnit) + _ = product.Descriptions.Select(x => + CreateField( + "Description_" + x.Key, + x.Value, + new CreateFieldOptions + { + Indexing = FieldIndexing.Search, + Storage = FieldStorage.No + })) }; + + StoreAllFields(FieldStorage.No); + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + Analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -{`public class Products_ByProductType_JS : AbstractJavaScriptIndexCreationTask + +```csharp +public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask { - public Products_ByProductType_JS() + public Products_ByLocalizedDescription_JS() { Maps = new HashSet { - @"map('Products', function (p) { + @"map('Products', function (product) { return { - _: createField(p.ProductType, p.PricePerUnit, - { indexing: 'Search', storage: true, termVector: null }) + _: Object.keys(product.Descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" }; + + Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions + { + Storage = FieldStorage.No + }; + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + Fields["Description_English"] = new IndexFieldOptions + { + Analyzer = "StopAnalyzer" + }; } } -`} - - - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' - .WhereEquals("Electronics", 23) - .ToList(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -from index 'Products/ByProductType' where Electronics = 23 -`} - +``` -## Example - list - - +#### Configure a fallback analyzer for all fields -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. - - - -* **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - public string Name \{ get; set; \} - - // For each element in this list, the VALUE of property 'PropName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - public List Attributes \{ get; set; \} -\} - -public class Attribute -\{ - public string PropName \{ get; set; \} - public string PropValue \{ get; set; \} -\} -`} - - - - - -{`// Sample document content -\{ -Name": "SomeName", -Attributes": [ - \{ - "PropName": "Color", - "PropValue": "Blue" - \}, - \{ - "PropName": "Width", - "PropValue": "10" - \}, - \{ - "PropName": "Length", - "PropValue": "20" - \}, - ... - -\} -`} - - - -* **The index**: - The below index will create a dynamic-index-field per item in the document's `Attributes` list. - New items added to the Attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's PropName **value**. - E.g., 'PropName' value `Width` will be a dynamic-index-field. - - - - -{`public class Attributes_ByName : AbstractIndexCreationTask + + +```csharp +public class Products_ByLocalizedDescription : AbstractIndexCreationTask { - public Attributes_ByName() + public Products_ByLocalizedDescription() { - Map = products => from a in products + Map = products => from product in products select new { - // Define the dynamic-index-fields by calling CreateField - // A dynamic-index-field will be generated for each item in the Attributes list - - // For each item, the field name will be the value of field 'PropName' - // The field terms will be derived from field 'PropValue' - _ = a.Attributes.Select(item => CreateField(item.PropName, item.PropValue)), - - // A regular index field can be defined as well: - Name = a.Name + _ = product.Descriptions.Select(x => + CreateField( + "Description_" + x.Key, + x.Value, + new CreateFieldOptions + { + Indexing = FieldIndexing.Search, + Storage = FieldStorage.No + })) }; + + StoreAllFields(FieldStorage.No); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + Analyze(Constants.Documents.Indexing.Fields.AllFields, "StopAnalyzer"); } } -`} - +``` - - -{`public class Attributes_ByName_JS : AbstractJavaScriptIndexCreationTask + +```csharp +public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask { - public Attributes_ByName_JS() + public Products_ByLocalizedDescription_JS() { Maps = new HashSet { - @"map('Products', function (p) { + @"map('Products', function (product) { return { - _: p.Attributes.map(item => createField(item.PropName, item.PropValue, - { indexing: 'Search', storage: true, termVector: null })), - Name: p.Name + _: Object.keys(product.Descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" }; + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions + { + Storage = FieldStorage.No, + Analyzer = "StopAnalyzer" + }; } } -`} - - - - -* **The query**: - * To get all documents matching a specific attribute property use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'Width' is a dynamic-index-field that was indexed from the Attributes list - .WhereEquals("Width", 10) - .ToList(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the Attributes list -from index 'Attributes/ByName' where Width = 10 -`} - +``` + + + + +### Which analyzer is used for the query term? + +The following applies when a `search()` query targets a **dynamic field**: + +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `AllFields` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `AllFields` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `Analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `Analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | + -## CreateField syntax + + + #### Syntax for LINQ-index: - - -{`object CreateField(string name, object value); + +```csharp +object CreateField(string name, object value); object CreateField(string name, object value, bool stored, bool analyzed); object CreateField(string name, object value, CreateFieldOptions options); -`} - +``` #### Syntax for JavaScript-index: - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` | Parameters | Type | Description | |----------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| | **fieldName** | `string` | Name of the dynamic-index-field | -| **fieldValue** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorate.Yes` | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorage.Yes` | | **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | | **options** | `CreateFieldOptions` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| | **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -540,11 +930,15 @@ object CreateField(string name, object value, CreateFieldOptions options); -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. + + + +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) + + \ No newline at end of file diff --git a/docs/indexes/content/_using-dynamic-fields-java.mdx b/docs/indexes/content/_using-dynamic-fields-java.mdx index 17f3cbe360..d54ef6bea8 100644 --- a/docs/indexes/content/_using-dynamic-fields-java.mdx +++ b/docs/indexes/content/_using-dynamic-fields-java.mdx @@ -2,479 +2,918 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -* For such dynamic data, you can define indexes with **dynamic-index-fields**. + -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. + -* Any value type can be indexed, string, number, date, etc. + + +### Example - index every field under an object -* An index definition can contain both dynamic-index-fields and regular-index-fields. + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. + -* In this page: +* **The document**: + + + ```java + public class Product { + private String id; - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) + // The KEYS under the attributes object will be dynamically indexed + // Fields added to this object after index creation time will also get indexed + private Map attributes; - + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "attributes": { + "color": "Red", + "size": 42 + } + } + ``` + -## Indexing documents fields KEYS +* **The index**: + + The following index will index any field under the `attributes` object from the document, + a dynamic-index-field will be created for each such field. + New fields added to the object after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `color` & `size` will become the actual dynamic-index-fields. + + + + ```java + public class Products_ByAttributeKey extends AbstractIndexCreationTask { + public Products_ByAttributeKey() { + + // Call 'CreateField' to generate dynamic-index-fields + // from the attributes object keys. + + // Using '_' is just a convention. + // Any other string can be used instead of '_'. + + // The actual field name will be 'item.Key' + // The actual field terms will be derived from 'item.Value' + map = "docs.Products.Select(p => new { " + + " _ = p.attributes.Select(item => " + + " this.CreateField(item.Key, item.Value)) " + + "})"; + } + } + ``` + + + ```java + public class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByAttributeKey_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + // The field name will be the key + // The field terms will be derived from the corresponding value + " _: Object.keys(p.attributes) " + + " .map(key => createField(key, p.attributes[key])) " + + " }; " + + "})" + )); + } + } + ``` + + -## Example - index any field under object +* **The query**: + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _size_ use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Products_ByAttributeKey.class) + // 'size' is a dynamic-index-field that was indexed from the attributes object + .whereEquals("size", 42) + .toList(); + ``` + + + ```sql + // 'size' is a dynamic-index-field that was indexed from the attributes object + from index 'Products/ByAttributeKey' where size = 42 + ``` + + + + + Dynamic index fields are queried with _DocumentQuery_ or _RQL_ because the field names (e.g. `size`, `color`) + are generated at indexing time and are not properties on the _Product_ class. + + Use the string-based `whereEquals("size", 42)` or RQL `where size = 42` instead. + - + -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + + +### Example - index every field + +The following example allows you to: + * Define an index on a collection **without** needing any common structure between the indexed documents. + * After the index is deployed, any new field added to any document in the collection will be indexed as well. + + + +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`public class Product \{ - private String id; - - // The KEYS under the attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - private Dictionary attributes; - - // get + set implementation ... -\} -`} - - - - -{`// Sample document content -\{ - "attributes": \{ - "color": "Red", - "size": 42 - \} -\} -`} - - + + + ```java + public class Product { + private String id; + + // All KEYS in the document will be dynamically indexed + // Fields added to the document after index creation time will also get indexed + private String firstName; + private String lastName; + private String title; + // ... + + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "firstName": "John", + "lastName": "Doe", + "title": "Engineer" + // ... + } + ``` + * **The index**: - The following index will index any field under the `attributes` object from the document, - a dynamic-index-field will be created for each such field. - New fields added to the object after index creation time will be dynamically indexed as well. + + The following index will index any field from the document, + a dynamic-index-field will be created for each field. + New fields added to the document after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + + The example uses an `AbstractJavaScriptIndexCreationTask` because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a map or collection. + + + + ```java + public class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByAnyField_JS() { + + // This will index EVERY FIELD under the top level of the document + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + " _: Object.keys(p).map(key => createField(key, p[key])) " + + " }; " + + "})" + )); + } + } + ``` + + - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - e.g. Keys `color` & `size` will become the actual dynamic-index-fields. +* **The query**: + + To get all documents with a specific _lastName_ use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Products_ByAnyField_JS.class) + // 'lastName' is a dynamic-index-field that was indexed from the document + .whereEquals("lastName", "Doe") + .toList(); + ``` + + + ```sql + // 'lastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where lastName = "Doe" + ``` + + + + + + - - - -{`public class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { - public Products_ByAttributeKey_JS() { - // Call 'createField' to generate dynamic-index-fields from the attributes object keys - // Using '_' is just a convention. Any other string can be used instead of '_' + - // The actual field name will be the key - // The actual field terms will be derived from p.attributes[key] - setMaps(Sets.newHashSet( - "map('Products', function (p) { " + - " return { " + - " _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], " + - " { indexing: 'Search', storage: false, termVector: null })) " + - " }; " + - "}) " - )); - } -} -`} - - - + + +### Example - basic -* **The query**: - * You can now query the generated dynamic-index fields. - Property `_` is Not queryable, it is only used in the index definition syntax. - * To get all documents with some 'size' use: - - - -{`List matchingDocuments = session - .query(Product.class, Products_ByAttributeKey_JS.class) - .whereEquals("size", 42) - .toList(); -`} - - - - -{`// 'size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey/JS' where size = 42 -`} - - - + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + -## Example - index any field +* **The document**: + + + ```java + public class Product { + private String id; - + // The VALUE of productType will be dynamically indexed + private String productType; + private int pricePerUnit; -* Define an index on a collection **without** needing any common structure between the indexed documents. -* After index is deployed, any new field added to the document will be indexed as well. + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "productType": "Electronics", + "pricePerUnit": 23 + } + ``` + - +* **The index**: + + The following index will index the **value** of document field 'productType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + + ```java + public class Products_ByProductType extends AbstractIndexCreationTask { + public Products_ByProductType() { + + // Call 'CreateField' to generate the dynamic-index-field. + // The field name will be the value of document field 'productType'. + // The field terms will be derived from document field 'pricePerUnit'. + map = "docs.Products.Select(p => new { " + + " _ = this.CreateField(p.productType, p.pricePerUnit) " + + "})"; + } + } + ``` + + + ```java + public class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByProductType_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + // The field name will be the value of document field 'productType' + // The field terms will be derived from document field 'pricePerUnit' + " _: createField(p.productType, p.pricePerUnit) " + + " }; " + + "})" + )); + } + } + ``` + + - -Consider if that is a true necessity, as indexing every single field can end up costing time and disk space. +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Products_ByProductType.class) + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + .whereEquals("Electronics", 23) + .toList(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + + +### Example - list + + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. * **The document**: - - -{`public class Product \{ - private String id; - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - private String firstName; - private String lastName; - private String title; - // ... - - // get + set implementation ... -\} -`} - - - - - -{`// Sample document content -\{ - "firstName": "John", - "lastName": "Doe", - "title": "Engineer", - // ... -\} -`} - - + + + ```java + public class Product { + private String id; + private String name; + + // For each element in this list, + // the VALUE of property 'propName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + private List attributes; + + // getters and setters ... + } -* **The index**: - The following index will index any field from the document, - a dynamic-index-field will be created for each field. - New fields added to the document after index creation time will be dynamically indexed as well. + public class Attribute { + private String propName; + private String propValue; - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "name": "SomeName", + "attributes": [ + { + "propName": "Color", + "propValue": "Blue" + }, + { + "propName": "Width", + "propValue": "10" + }, + { + "propName": "Length", + "propValue": "20" + } + // ... + ] + } + ``` + - - - -{`public class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { - public Products_ByAnyField_JS() { +* **The index**: - // This will index EVERY FIELD under the top level of the document - setMaps(Sets.newHashSet( - "map('Products', function (p) { " + - " return { " + - " _: Object.keys(p).map(key => createField(key, p[key], " + - " { indexing: 'Search', storage: true, termVector: null })) " + - " }; " + - "}) " - )); + The following index will create a dynamic-index-field per item in the document's `attributes` list. + New items added to the attributes list after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the item's propName **value**. + E.g., 'propName' value `Width` will be a dynamic-index-field. + + + + ```java + public class Attributes_ByName extends AbstractIndexCreationTask { + public Attributes_ByName() { + + // Define the dynamic-index-fields by calling 'CreateField'. + // A dynamic-index-field will be generated for each item in 'attributes'. + + // For each item, the field name will be the value of field 'propName' + // The field terms will be derived from field 'propValue'. + // A regular index field (Name) is defined as well. + map = "docs.Products.Select(p => new { " + + " _ = p.attributes.Select(item => " + + " this.CreateField(item.propName, item.propValue)), " + + " Name = p.name " + + "})"; + } } -} -`} - - - + ``` + + + ```java + public class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask { + public Attributes_ByName_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + // For each item, + // the field name will be the value of field 'propName'. + // The field terms will be derived from field 'propValue'. + " _: p.attributes.map(item => " + + " createField(item.propName, item.propValue)), " + + // A regular index field can be defined as well: + " Name: p.name " + + " }; " + + "})" + )); + } + } + ``` + + * **The query**: - * To get all documents with some 'lastName' use: - - - -{`List matchingDocuments = session - .query(Product.class, Products_ByAnyField_JS.class) - .whereEquals("lastName", "Doe") - .toList(); -`} - - - - -{`// 'lastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where lastName = "Doe" -`} - - - + + To get all documents matching a specific attribute property use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Attributes_ByName.class) + // 'Width' is a dynamic-index-field that was indexed from the attributes list + .whereEquals("Width", 10) + .toList(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `indexing: 'Search'` in the `createField` options object (`AbstractJavaScriptIndexCreationTask`) +or `FieldIndexing.Search` in the `CreateFieldOptions` (`AbstractIndexCreationTask`). +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. -## Indexing documents fields VALUES + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). -## Example - basic +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. - + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. -* For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---index-a-list-of-properties) below. + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: -* **The document**: - - -{`public class Product \{ - private String id; - - // The VALUE of productType will be dynamically indexed - private String productType; - private int pricePerUnit; - - // get + set implementation ... -\} -`} - - + + ```java + public class Product { + private String id; - - -{`// Sample document content -\{ - "productType": "Electronics", - "pricePerUnit": 23 -\} -`} - - + private Map descriptions; -* **The index**: - The following index will index the **value** of document field 'productType'. + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `descriptions` map. + For example, the key `English` creates the dynamic field `Description_English`. + + Each generated dynamic field is indexed for **full-text search**. + + + + ```java + public class Products_ByLocalizedDescription extends AbstractIndexCreationTask { + + private static final String USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription() { + map = "docs.Products.Select(product => new { " + + " _ = product.descriptions.Select(x => " + + " this.CreateField( " + + " \"Description_\" + x.Key, " + + " x.Value, " + + " new CreateFieldOptions { " + + // Index each generated dynamic field for FTS + " Indexing = FieldIndexing.Search, " + + " Storage = FieldStorage.No " + + " })) " + + "})"; + + storeAllFields(FieldStorage.NO); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + getConfiguration().put(USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS, "true"); + } + } + ``` + + + ```java + public class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + + private static final String USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (product) { " + + " return { " + + " _: Object.keys(product.descriptions) " + + " .map(function (key) { " + + " return createField( " + + " 'Description_' + key, " + + " product.descriptions[key], " + + " { " + + // Index each generated dynamic field for FTS + " indexing: 'Search', " + + " storage: false " + + " }); " + + " }) " + + " }; " + + "})" + )); + + Map fields = new HashMap<>(); + + IndexFieldOptions allFieldsOptions = new IndexFieldOptions(); + allFieldsOptions.setStorage(FieldStorage.NO); + + fields.put(Constants.Documents.Indexing.Fields.ALL_FIELDS, allFieldsOptions); + setFields(fields); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + getConfiguration().put(USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS, "true"); + } + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```java + List results = session.advanced() + .documentQuery(Product.class, Products_ByLocalizedDescription.class) + .search("Description_English", "north wind") + .toList(); + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + - This value will be the dynamic-index-field name on which you can query. - e.g. Field value `Electronics` will be the dynamic-index-field. +--- - - - -{`public class Products_ByProductType extends AbstractIndexCreationTask { - public Products_ByProductType() { + + +### Configuring analyzers for dynamic fields + +The `CreateFieldOptions` (and the JavaScript `createField` options) do **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: - // The field name will be the value of document field 'productType' - // The field terms will be derived from document field 'pricePerUnit' - map = "docs.Products.Select(p => new { " + - " _ = this.CreateField(p.productType, p.pricePerUnit) " + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field + + + +```java +public class Products_ByLocalizedDescription extends AbstractIndexCreationTask { + public Products_ByLocalizedDescription() { + map = "docs.Products.Select(product => new { " + + " _ = product.descriptions.Select(x => " + + " this.CreateField( " + + " \"Description_\" + x.Key, " + + " x.Value, " + + " new CreateFieldOptions { " + + " Indexing = FieldIndexing.Search, " + + " Storage = FieldStorage.No " + + " })) " + "})"; + + storeAllFields(FieldStorage.NO); + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`List matchingDocuments = session - .query(Product.class, Products_ByProductType.class) - .whereEquals("Electronics", 23) - .toList(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'productType' -from index 'Products/ByProductType' where Electronics = 23 -`} - - - - -## Example - list + +```java +public class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByLocalizedDescription_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (product) { " + + " return { " + + " _: Object.keys(product.descriptions) " + + " .map(function (key) { " + + " return createField( " + + " 'Description_' + key, " + + " product.descriptions[key], " + + " { " + + " indexing: 'Search', " + + " storage: false " + + " }); " + + " }) " + + " }; " + + "})" + )); - + Map fields = new HashMap<>(); -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. + IndexFieldOptions allFieldsOptions = new IndexFieldOptions(); + allFieldsOptions.setStorage(FieldStorage.NO); + fields.put(Constants.Documents.Indexing.Fields.ALL_FIELDS, allFieldsOptions); - + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + IndexFieldOptions englishOptions = new IndexFieldOptions(); + englishOptions.setAnalyzer("StopAnalyzer"); + fields.put("Description_English", englishOptions); -* **The document**: - - -{`public class Product \{ - private String id; - private String name; - - // For each element in this list, the VALUE of property 'propName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - private List attributes; - - // get + set implementation ... -\} - -public class Attribute \{ - private String propName; - private String propValue; - - // get + set implementation ... -\} -`} - + setFields(fields); + } +} +``` + - - -{`// Sample document content -\{ -name": "SomeName", -attributes": [ - \{ - "propName": "Color", - "propValue": "Blue" - \}, - \{ - "propName": "Width", - "propValue": "10" - \}, - \{ - "propName": "Length", - "propValue": "20" - \}, - ... - -\} -`} - - +#### Configure a fallback analyzer for all fields + + + +```java +public class Products_ByLocalizedDescription extends AbstractIndexCreationTask { + public Products_ByLocalizedDescription() { + map = "docs.Products.Select(product => new { " + + " _ = product.descriptions.Select(x => " + + " this.CreateField( " + + " \"Description_\" + x.Key, " + + " x.Value, " + + " new CreateFieldOptions { " + + " Indexing = FieldIndexing.Search, " + + " Storage = FieldStorage.No " + + " })) " + + "})"; -* **The index**: - The following index will create a dynamic-index-field per item in the document's `attributes` list. - New items added to the attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's propName **value**. - E.g., 'propName' value `Width` will be a dynamic-index-field. - - - - -{`public class Attributes_ByName extends AbstractIndexCreationTask { - public Attributes_ByName() { - - // For each attribute item, the field name will be the value of field 'propName' - // The field terms will be derived from field 'propValue' - // A regular-index-field (Name) is defined as well - map = - "docs.Products.Select(p => new { " + - " _ = p.attributes.Select(item => this.CreateField(item.propName, item.propValue)), " + - " Name = p.name " + - "})"; + storeAllFields(FieldStorage.NO); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + analyze(Constants.Documents.Indexing.Fields.ALL_FIELDS, "StopAnalyzer"); } } -`} - +``` - + +```java +public class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByLocalizedDescription_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (product) { " + + " return { " + + " _: Object.keys(product.descriptions) " + + " .map(function (key) { " + + " return createField( " + + " 'Description_' + key, " + + " product.descriptions[key], " + + " { " + + " indexing: 'Search', " + + " storage: false " + + " }); " + + " }) " + + " }; " + + "})" + )); -* **The query**: - * To get all documents matching a specific attribute property use: - - - -{`List matchingDocuments = session - .query(Product.class, Attributes_ByName.class) - .whereEquals("Width", 10) - .toList(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the attributes list -from index 'Attributes/ByName' where Width = 10 -`} - + Map fields = new HashMap<>(); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + IndexFieldOptions allFieldsOptions = new IndexFieldOptions(); + allFieldsOptions.setStorage(FieldStorage.NO); + allFieldsOptions.setAnalyzer("StopAnalyzer"); + fields.put(Constants.Documents.Indexing.Fields.ALL_FIELDS, allFieldsOptions); + + setFields(fields); + } +} +``` + + + + +### Which analyzer is used for the query term? +The following applies when a `search()` query targets a **dynamic field**: -## CreateField syntax +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | -#### Syntax for LINQ-index: + - - -{`object CreateField(string name, object value); + + + + +#### Syntax for `AbstractIndexCreationTask` + + +```csharp +object CreateField(string name, object value); object CreateField(string name, object value, bool stored, bool analyzed); object CreateField(string name, object value, CreateFieldOptions options); -`} - +``` -#### Syntax for JavaScript-index: +#### Syntax for `AbstractJavaScriptIndexCreationTask` - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` -| Parameters | Type | Description | -|----------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **fieldName** | `string` | Name of the dynamic-index-field | -| **fieldValue** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorate.Yes` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | +| Parameters | Type | Description | +|----------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorage.Yes` | +| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | | **options** | `CreateFieldOptions` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| | **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +* The `CreateField` syntax (and the `CreateFieldOptions` shape) above describes what is available inside the server-side `map` string of an `AbstractIndexCreationTask`, + since the Java client sends the `map` source to the server for compilation. + The JavaScript `createField` function is the equivalent for `AbstractJavaScriptIndexCreationTask` map sources. + * All above examples have used the character `_` in the dynamic-index-field definition. However, using `_` is just a convention. Any other string can be used instead. * This property is Not queryable, it is only used in the index definition syntax. - The actual dynamic-index-fields that are generated are defined by the `CreateField` method. + The actual dynamic-index-fields that are generated are defined by the `CreateField` method (or the `createField` JS function). + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. - +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/docs/indexes/content/_using-dynamic-fields-nodejs.mdx b/docs/indexes/content/_using-dynamic-fields-nodejs.mdx index b5df2df4a7..c14184c2b5 100644 --- a/docs/indexes/content/_using-dynamic-fields-nodejs.mdx +++ b/docs/indexes/content/_using-dynamic-fields-nodejs.mdx @@ -2,533 +2,710 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. - -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. - -* In this page: - - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`createField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -## Indexing documents fields KEYS + - -#### Example - index any field under object -The following allows you to: - -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. -**The document**: - - -{`class Product \{ - constructor(id, attributes) \{ - this.id = id; - - // The KEYS under the attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - this.attributes = attributes; - \} -\} -`} - - + + +### Example - index every field under an object - - -{`// Sample document content -\{ - "attributes": \{ - "color": "Red", - "size": 42 - \} -\} -`} - - + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. + -**The index**: +* **The document**: + + + ```js + class Product { + constructor(id, attributes) { + this.id = id; + + // The KEYS under the attributes object will be dynamically indexed + // Fields added to this object after index creation time will also get indexed + this.attributes = attributes; + } + } + ``` + + + + ```json + // Sample document content + { + "attributes": { + "color": "Red", + "size": 42 + } + } + ``` + -* The following index will index any field under the `attributes` object from the document, +* **The index**: + + The following index will index any field under the `attributes` object from the document, a dynamic-index-field will be created for each such field. - New fields added to the object after index creation time will be dynamically indexed as well. - -* The actual dynamic-index-field name on which you can query will be the attribute field **key**. - e.g. Keys `color` & `size` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { - constructor() { - super(); - - const { createField } = this.mapUtils(); - - this.map("Products", p => { - return { - // Call 'createField' to generate dynamic-index-fields from the attributes object keys - // Using '_' is just a convention. Any other string can be used instead of '_' - - // The actual field name will be the key - // The actual field terms will be derived from p.attributes[key] - _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], { - indexing: "Search", - storage: false, - termVector: null - })) - }; - }); + New fields added to the object after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `color` & `size` will become the actual dynamic-index-fields. + + + ```js + class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + this.map("Products", p => { + return { + // Call 'createField' to generate dynamic-index-fields + // from the attributes object keys. + + // Using '_' is just a convention. + // Any other string can be used instead of '_' + + // The actual field name will be the key + // The actual field terms will be derived from p.attributes[key] + _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key])) + }; + }); + } } -} -`} - - - - -**The query**: - -* You can now query the generated dynamic-index fields. - Property `_` is Not queryable, it is only used in the index definition syntax. - -* To get all documents with some 'size' use: - - - - -{`const matchingDocuments = session.query({indexName: 'Products_ByAttributeKey'}) - // 'size' is a dynamic-index-field that was indexed from the attributes object - .whereEquals('size', 42) - .all(); -`} - - - - -{`// 'size' is a dynamic-index-field that was indexed from the attributes object -from index 'Products/ByAttributeKey' where size = 42 -`} - - - - - + ``` + + +* **The query**: + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _size_ use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Products/ByAttributeKey/JS" }) + // 'size' is a dynamic-index-field that was indexed from the attributes object + .whereEquals("size", 42) + .all(); + ``` + + + ```sql + // 'size' is a dynamic-index-field that was indexed from the attributes object + from index 'Products/ByAttributeKey/JS' where size = 42 + ``` + + + + + + + +### Example - index every field -#### Example - index any field -The following allows you to: - -* Define an index on a collection **without** needing any common structure between the indexed documents. -* After index is deployed, any new field added to the document will be indexed as well. - +The following example allows you to: + * Define an index on a collection **without** needing any common structure between the indexed documents. + * After the index is deployed, any new field added to any document in the collection will be indexed as well. + + - -Consider if that is a true necessity, as indexing every single field can end up costing time and disk space. - +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. -**The document**: - - -{`class Product \{ - constructor(id, firstName, lastName, title) \{ - this.id = id; - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - this.firstName = firstName; - this.lastName = lastName; - this.title = title; - // ... - \} -\} -`} - - - - - -{`// Sample document content -\{ - "firstName": "John", - "lastName": "Doe", - "title": "Engineer", - // ... -\} -`} - - -**The index**: +* **The document**: + + + ```js + class Product { + constructor(id, firstName, lastName, title) { + this.id = id; + + // All KEYS in the document will be dynamically indexed + // Fields added to the document after index creation time will also get indexed + this.firstName = firstName; + this.lastName = lastName; + this.title = title; + // ... + } + } + ``` + + + + ```json + // Sample document content + { + "firstName": "John", + "lastName": "Doe", + "title": "Engineer", + // ... + } + ``` + -* The following index will index any field from the document, +* **The index**: + + The following index will index any field from the document, a dynamic-index-field will be created for each field. - New fields added to the document after index creation time will be dynamically indexed as well. - -* The actual dynamic-index-field name on which you can query will be the field **key**. - e.g. Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + New fields added to the document after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + + + ```js + class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + // This will index EVERY FIELD under the top level of the document + this.map("Products", p => { + return { + _: Object.keys(p).map(key => createField(key, p[key])) + }; + }); + } + } + ``` + + +* **The query**: + + To get all documents with a specific _lastName_ use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Products/ByAnyField/JS" }) + // 'lastName' is a dynamic-index-field that was indexed from the document + .whereEquals("lastName", "Doe") + .all(); + ``` + + + ```sql + // 'lastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where lastName = "Doe" + ``` + + + + + + + + + + + +### Example - basic + + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + - - - -{`class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { - constructor () { - super(); +* **The document**: + + + ```js + class Product { + constructor(id, productType, pricePerUnit) { + this.id = id; + + // The VALUE of productType will be dynamically indexed + this.productType = productType; + this.pricePerUnit = pricePerUnit; + } + } + ``` + + + + ```json + // Sample document content + { + "productType": "Electronics", + "pricePerUnit": 23 + } + ``` + + +* **The index**: + + The following index will index the **value** of document field 'productType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + ```js + class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + this.map("Products", p => { + return { + // The field name will be the value of document field 'productType' + // The field terms will be derived from document field 'pricePerUnit' + _: createField(p.productType, p.pricePerUnit) + }; + }); + } + } + ``` + + +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Products/ByProductType/JS" }) + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + .whereEquals("Electronics", 23) + .all(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + from index 'Products/ByProductType/JS' where Electronics = 23 + ``` + + + + + + + +### Example - list - const { createField } = this.mapUtils(); + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. + - this.map("Products", p => { - return { - // This will index EVERY FIELD under the top level of the document - _: Object.keys(p).map(key => createField(key, p[key], { - indexing: "Search", - storage: true, - termVector: null - })) - }; - }); +* **The document**: + + + ```js + class Product { + constructor(id, name, attributes) { + this.id = id; + this.name = name; + + // For each element in this list, + // the VALUE of property 'propName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + this.attributes = attributes; + } } -} -`} - - - -**The query**: + class Attribute { + constructor(propName, propValue) { + this.propName = propName; + this.propValue = propValue; + } + } + ``` + + + + ```json + // Sample document content + { + "name": "SomeName", + "attributes": [ + { + "propName": "Color", + "propValue": "Blue" + }, + { + "propName": "Width", + "propValue": "10" + }, + { + "propName": "Length", + "propValue": "20" + }, + ... + ] + } + ``` + + +* **The index**: + + The following index will create a dynamic-index-field per item in the document's `attributes` list. + New items added to the attributes list after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the item's propName **value**. + E.g., 'propName' value `Width` will be a dynamic-index-field. + + + ```js + class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + this.map("Products", p => { + return { + // Define the dynamic-index-fields by calling 'createField' + // A dynamic-index-field will be generated for each item in 'attributes' + + // For each item, the field name will be the value of field 'propName' + // The field terms will be derived from field 'propValue' + _: p.attributes.map(item => + createField(item.propName, item.propValue)), + + // A regular index field can be defined as well: + name: p.name + }; + }); + } + } + ``` + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Attributes/ByName/JS" }) + // 'Width' is a dynamic-index-field that was indexed from the attributes list + .whereEquals("Width", 10) + .all(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the attributes list + from index 'Attributes/ByName/JS' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `indexing: "Search"` in the `createField` options. + +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. + + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. + + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. -* To get all documents with some 'lastName' use: + + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```js + class Product { + constructor(id, descriptions) { + this.id = id; + this.descriptions = descriptions; + } + } + ``` + + + + ```json + // Sample document content + { + "descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. + + Each generated dynamic field is indexed for **full-text search**. + + + ```js + class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + const useSearchAnalyzerForDynamicFields = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + this.map("Products", product => { + return { + _: Object.keys(product.descriptions).map(key => + createField( + "Description_" + key, + product.descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: "Search", + storage: false + })) + }; + }); + + this.storeAllFields("No"); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + this.configuration[useSearchAnalyzerForDynamicFields] = "true"; + } + } + ``` + - - - -{`const matchingDocuments = session.query({ indexName: 'Products_ByAnyField_JS' }) - // 'lastName' is a dynamic-index-field that was indexed from the document - .whereEquals('lastName', 'Doe') - .all(); -`} - - - - -{`// 'lastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where lastName = "Doe" -`} - - - +* **Full-text search query**: - + Query the generated dynamic field by its field name. + In this example, the query targets the generated field `Description_English`. + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). -## Indexing documents fields VALUES + + + ```js + const results = await session + .query({ indexName: "Products/ByLocalizedDescription/JS" }) + .search("Description_English", "north wind") + .all(); + ``` + - -#### Example - basic -This example shows: - -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. -* For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---index-a-list-of-properties) below. -**The document**: - - -{`class Product \{ - constructor(id, productType, pricePerUnit) \{ - this.id = id; - - // The VALUE of productType will be dynamically indexed - this.productType = productType; - this.pricePerUnit = pricePerUnit; - \} -\} -`} - - + + ```sql + from index "Products/ByLocalizedDescription/JS" + where search(Description_English, "north wind") + ``` + + - - -{`// Sample document content -\{ - "productType": "Electronics", - "pricePerUnit": 23 -\} -`} - - +--- -**The index**: + + +### Configuring analyzers for dynamic fields -* The following index will index the **value** of document field 'productType'. +The `createField` options object does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. -* This value will be the dynamic-index-field name on which you can query. - e.g. Field value `Electronics` will be the dynamic-index-field. +--- - - - -{`class Products_ByProductType extends AbstractCsharpIndexCreationTask { - constructor () { - super(); +#### Configure analyzer for a specific dynamic field - // The field name will be the value of document field 'productType' - // The field terms will be derived from document field 'pricePerUnit' - this.map = "docs.Products.Select(p => new { " + - " _ = this.CreateField(p.productType, p.pricePerUnit) " + - "})"; - } -} -`} - - - - -{`class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask { - constructor () { + +```js +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { super(); const { createField } = this.mapUtils(); - this.map("Products", p => { + this.map("Products", product => { return { - _: [ - // The field name will be the value of document field 'productType' - // The field terms will be derived from document field 'pricePerUnit' - createField(p.productType, p.pricePerUnit, { - indexing: "Search", - storage: false, - termVector: null - }) - ] + _: Object.keys(product.descriptions).map(key => + createField( + "Description_" + key, + product.descriptions[key], + { + indexing: "Search", + storage: false + })) }; }); - } -} -`} - - - - -**The query**: - -* To get all documents of some product type having a specific price per unit use: - - - - -{`const matchingDocuments = session.query({ indexName: 'Products_ByProductType' }) - // 'Electronics' is the dynamic-index-field that was indexed from document field 'productType' - .whereEquals('Electronics', 23) - .all(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'productType' -from index 'Products/ByProductType' where Electronics = 23 -`} - - - - - - -#### Example - list -The following allows you to: - -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. -**The document**: - - -{`class Product \{ - constructor(id, name, attributes) \{ - this.id = id; - this.name = name; - - // For each element in this list, the VALUE of property 'propName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - this.attributes = attributes; - \} -\} - -class Attribute \{ - constructor(propName, propValue) \{ - this.propName = propName; - this.propValue = propValue; - \} -\} -`} - - - - - -{`// Sample document content -\{ - "name": "SomeName", - "attributes": [ - \{ - "propName": "Color", - "propValue": "Blue" - \}, - \{ - "propName": "Width", - "propValue": "10" - \}, - \{ - "propName": "Length", - "propValue": "20" - \}, - ... - ] -\} -`} - - - -**The index**: - -* The following index will create a dynamic-index-field per item in the document's `attributes` list. - New items added to the attributes list after index creation time will be dynamically indexed as well. - -* The actual dynamic-index-field name on which you can query will be the item's propName **value**. - e.g. 'propName' value `Width` will be a dynamic-index-field. - - - - -{`class Attributes_ByName extends AbstractCsharpIndexCreationTask -{ - constructor () { - super(); + this.storeAllFields("No"); - // For each attribute item, the field name will be the value of field 'propName' - // The field terms will be derived from field 'propValue' - // A regular-index-field (Name) is defined as well - this.map = - "docs.Products.Select(p => new { " + - " _ = p.attributes.Select(item => this.CreateField(item.propName, item.propValue)), " + - " Name = p.name " + - "})"; + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + this.analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -{`class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask { - constructor () { + +#### Configure a fallback analyzer for all fields + + +```js +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { super(); const { createField } = this.mapUtils(); - this.map("Products", p => { + this.map("Products", product => { return { - // For each item, the field name will be the value of field 'propName' - // The field terms will be derived from field 'propValue' - _: p.attributes.map(item => createField(item.propName, item.propValue, { - indexing: "Search", - storage: true, - termVector: null - })), - - // A regular-index-field can be defined as well: - Name: p.name + _: Object.keys(product.descriptions).map(key => + createField( + "Description_" + key, + product.descriptions[key], + { + indexing: "Search", + storage: false + })) }; }); - } -} -`} - - - - -**The query**: -* To get all documents matching a specific attribute property use: + this.storeAllFields("No"); - - - -{`const matchingDocuments = session.query({ indexName: 'Attributes/ByName' }) - // 'Width' is a dynamic-index-field that was indexed from the attributes list - .whereEquals('Width', 10) - .all(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the attributes list -from index 'Attributes/ByName' where Width = 10 -`} - + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + this.analyze(CONSTANTS.Documents.Indexing.Fields.ALL_FIELDS, "StopAnalyzer"); + } +} +``` - - - + + + + +### Which analyzer is used for the query term? -## CreateField syntax +The following applies when a `search()` query targets a **dynamic field**: -#### Syntax for LINQ-index: +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `this.analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `this.analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | - - -{`object CreateField(string name, object value); + -object CreateField(string name, object value, bool stored, bool analyzed); + -object CreateField(string name, object value, CreateFieldOptions options); -`} - - + -#### Syntax for JavaScript-index: +#### Syntax: - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` -| Parameters | Type | Description | -|----------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **fieldName** | `string` | Name of the dynamic-index-field | -| **fieldValue** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorate.Yes` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | -| **options** | `CreateFieldOptions` | Dynamic-index-field options | +| Parameters | Type | Description | +|----------------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **options** | `object` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| -| **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| options object | | | +|-----------------|--------------------|-----------------------------------------------------------------------------------| +| **storage** | `boolean` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | +| **indexing** | `FieldIndexing` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **termVector** | `FieldTermVector` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -536,21 +713,19 @@ object CreateField(string name, object value, CreateFieldOptions options); However, using `_` is just a convention. Any other string can be used instead. * This property is Not queryable, it is only used in the index definition syntax. - The actual dynamic-index-fields that are generated are defined by the `CreateField` method. + The actual dynamic-index-fields that are generated are defined by the `createField` method. + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. - +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/docs/indexes/content/_using-dynamic-fields-php.mdx b/docs/indexes/content/_using-dynamic-fields-php.mdx index 09c58cc9fc..6992310715 100644 --- a/docs/indexes/content/_using-dynamic-fields-php.mdx +++ b/docs/indexes/content/_using-dynamic-fields-php.mdx @@ -2,537 +2,969 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. + -* In this page: + - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) + + +### Example - index every field under an object + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. -## Indexing documents fields KEYS - -## Example - index any field under object +* **The document**: + + + ```php + use Ds\Map as DSMap; - + class Product + { + public ?string $id = null; -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + // The KEYS under the attributes object will be dynamically indexed. + // Fields added to this object after index creation time will also get indexed. + public ?DSMap $attributes = null; - - -* **The document**: - - -{`use Ds\\Map as DSMap; - -class Product -\{ - private ?string $id = null; - - // The KEYS under the Attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - public ?DSMap $attributes = null; -\} -`} - - - - -{`// Sample document content -\{ - "Attributes": \{ - "Color": "Red", - "Size": 42 - \} -\} -`} - - + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "attributes": { + "Color": "Red", + "Size": 42 + } + } + ``` + * **The index**: - The below index will index any field under the `Attributes` object from the document, + + The following index will index any field under the `attributes` object from the document, a dynamic-index-field will be created for each such field. New fields added to the object after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. - - - -{`class Products_ByAttributeKey extends AbstractIndexCreationTask -{ - public function __construct() + + + ```php + class Products_ByAttributeKey extends AbstractIndexCreationTask { - parent::__construct(); - - $this->map = "from p in docs.Products select new {" . - "_ = p.attributes.Select(item => CreateField(item.Key, item.Value))" . - "}"; + public function __construct() + { + parent::__construct(); + + // Call 'CreateField' to generate dynamic-index-fields + // from the attributes object keys. + // + // Using '_' is just a convention. + // Any other string can be used instead of '_'. + // + // The actual field name will be 'item.Key'. + // The actual field terms will be derived from 'item.Value'. + $this->map = + "from p in docs.Products " . + "select new { " . + " _ = p.attributes.Select(item => CreateField(item.Key, item.Value)) " . + "}"; + } } -} -`} - - - - -{`class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask -{ - public function __construct() + ``` + + + ```php + class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { - parent::__construct(); - - $this->setMaps([ - "map('Products', function (p) { " . - " return { " . - " _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], " . - " { indexing: 'Search', storage: false, termVector: null })) " . - " }; " . - "}) " - ]); + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (p) { + return { + // The field name will be the key + // The field terms will be derived from the corresponding value + _: Object.keys(p.attributes) + .map(key => createField(key, p.attributes[key], null)) + }; + })" + ]); + } } -} -`} - - - - + ``` + + * **The query**: - * You can now query the generated dynamic-index fields. - * To get all documents with some 'size' use: - - - -{`$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Products_ByAttributeKey::class) - // 'Size' is a dynamic-index-field that was indexed from the Attributes object - ->whereEquals("Size", 42) - ->toList(); -`} - - - - -{`// 'Size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey' where Size = 42 -`} - - - - -## Example - index any field - - + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _Size_ use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Products_ByAttributeKey::class) + // 'Size' is a dynamic-index-field that was indexed from the attributes object + ->whereEquals("Size", 42) + ->toList(); + ``` + + + ```sql + // 'Size' is a dynamic-index-field that was indexed from the attributes object + from index 'Products/ByAttributeKey' where Size = 42 + ``` + + + + + Dynamic index fields are queried with _documentQuery_ or _RQL_ because the field names (e.g. `Size`, `Color`) + are generated at indexing time and are not properties on the _Product_ class. + + Use the string-based `whereEquals("Size", 42)` or RQL `where Size = 42` instead. + + + + + + +### Example - index every field + +The following example allows you to: * Define an index on a collection **without** needing any common structure between the indexed documents. - * After index is deployed, any new field added to the document will be indexed as well. - + * After the index is deployed, any new field added to any document in the collection will be indexed as well. - + -Consider whether this is really necessary, as indexing every single field can end up costing time and disk space. +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`class Product -\{ - private ?string $id = null; - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - public ?string $firstName = null; - public ?string $lastName = null; - public ?string $title = null; - - // ... getters and setters -\} -`} - - + + + ```php + class Product + { + public ?string $id = null; - - -{`// Sample document content - \{ - "FirstName": "John", - "LastName": "Doe", - "Title": "Engineer", + // All KEYS in the document will be dynamically indexed. + // Fields added to the document after index creation time will also get indexed. + public ?string $firstName = null; + public ?string $lastName = null; + public ?string $title = null; // ... -\} -`} - - + + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "firstName": "John", + "lastName": "Doe", + "title": "Engineer" + // ... + } + ``` + * **The index**: - The below index will index any field from the document, + + The following index will index any field from the document, a dynamic-index-field will be created for each field. New fields added to the document after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask -{ - public function __construct() + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + + The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection. + + + + ```php + class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { - parent::__construct(); - - // This will index EVERY FIELD under the top level of the document - $this->setMaps([ - "map('Products', function (p) { - return { - _: Object.keys(p).map(key => createField(key, p[key], - { indexing: 'Search', storage: true, termVector: null })) - } - })" - ]); + public function __construct() + { + parent::__construct(); + + // This will index EVERY FIELD under the top level of the document + $this->setMaps([ + "map('Products', function (p) { + return { + _: Object.keys(p).map(key => createField(key, p[key], null)) + }; + })" + ]); + } } -} -`} - - - + ``` + + * **The query**: - * To get all documents with some 'LastName' use: - - - -{`$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Products_ByAnyField_JS::class) - // 'LastName' is a dynamic-index-field that was indexed from the document - ->whereEquals("LastName", "Doe") - ->toList(); -`} - - - - -{`// 'LastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where LastName = "Doe" -`} - - - + + To get all documents with a specific _lastName_ use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Products_ByAnyField_JS::class) + // 'lastName' is a dynamic-index-field that was indexed from the document + ->whereEquals("lastName", "Doe") + ->toList(); + ``` + + + ```sql + // 'lastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where lastName = "Doe" + ``` + + + + + + + + + + + +### Example - basic + + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + +* **The document**: + + + ```php + class Product + { + public ?string $id = null; + // The VALUE of productType will be dynamically indexed + public ?string $productType = null; + public ?int $pricePerUnit = null; -## Indexing documents fields VALUES + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "productType": "Electronics", + "pricePerUnit": 23 + } + ``` + -## Example - basic +* **The index**: + + The following index will index the **value** of document field 'productType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. - + + + ```php + class Products_ByProductType extends AbstractIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + // Call 'CreateField' to generate the dynamic-index-fields. + // The field name will be the value of document field 'productType'. + // The field terms will be derived from document field 'pricePerUnit'. + $this->map = + "from p in docs.Products " . + "select new { " . + " _ = CreateField(p.productType, p.pricePerUnit) " . + "}"; + } + } + ``` + + + ```php + class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (p) { + return { + // The field name will be the value of document field 'productType' + // The field terms will be derived from document field 'pricePerUnit' + _: createField(p.productType, p.pricePerUnit, null) + }; + })" + ]); + } + } + ``` + + -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Products_ByProductType::class) + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + ->whereEquals("Electronics", 23) + ->toList(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + + +### Example - list + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. * **The document**: - - -{`class Product -\{ - public ?string $id = null; - - // The VALUE of ProductType will be dynamically indexed - public ?string $productType = null; - public ?int $pricePerUnit = null; - - // ... getters and setters -\} -`} - - + + + ```php + class Product + { + public ?string $id = null; + public ?string $name = null; - - -{`// Sample document content -\{ - "ProductType": "Electronics", - "PricePerUnit": 23 -\} -`} - - + // For each element in this list, + // the VALUE of property 'propName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + public ?AttributeList $attributes = null; + + // ... getters and setters + } + + class Attribute + { + public ?string $propName = null; + public ?string $propValue = null; + + // ... getters and setters + } + + class AttributeList extends TypedList + { + protected function __construct() + { + parent::__construct(Attribute::class); + } + } + ``` + + + + ```json + // Sample document content + { + "name": "SomeName", + "attributes": [ + { + "propName": "Color", + "propValue": "Blue" + }, + { + "propName": "Width", + "propValue": "10" + }, + { + "propName": "Length", + "propValue": "20" + } + // ... + ] + } + ``` + * **The index**: - The below index will index the **value** of document field 'ProductType'. + + The following index will create a dynamic-index-field per item in the document's `attributes` list. + New items added to the attributes list after index creation time will be dynamically indexed as well. - This value will be the dynamic-index-field name on which you can query. - E.g., Field value `Electronics` will be the dynamic-index-field. + The actual dynamic-index-field name on which you can query will be the item's propName **value**. + E.g., 'propName' value `Width` will be a dynamic-index-field. + + + + ```php + class Attributes_ByName extends AbstractIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + // Define the dynamic-index-fields by calling 'CreateField'. + // A dynamic-index-field will be generated for each item in 'attributes'. + // + // For each item, the field name will be the value of field 'propName'. + // The field terms will be derived from field 'propValue'. + $this->map = + "from p in docs.Products " . + "select new { " . + " _ = p.attributes.Select(item => CreateField(item.propName, item.propValue)), " . + " Name = p.name " . + "}"; + } + } + ``` + + + ```php + class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (p) { + return { + // For each item, + // the field name will be the value of field 'propName' + // The field terms will be derived from field 'propValue' + _: p.attributes.map(item => createField( + item.propName, item.propValue, null)), + + // A regular index field can be defined as well: + Name: p.name + }; + })" + ]); + } + } + ``` + + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Attributes_ByName::class) + // 'Width' is a dynamic-index-field that was indexed from the attributes list + ->whereEquals("Width", 10) + ->toList(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `FieldIndexing::search()` in `CreateFieldOptions`. + +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. + + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. + + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + + + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```php + use Ds\Map as DSMap; + + class Product + { + public ?string $id = null; + public ?DSMap $descriptions = null; + + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. - + Each generated dynamic field is indexed for **full-text search**. + + + + ```php + class Products_ByLocalizedDescription extends AbstractIndexCreationTask + { + private const USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public function __construct() + { + parent::__construct(); + + $this->map = + "from product in docs.Products " . + "select new { " . + " _ = product.descriptions.Select(x => " . + " CreateField(" . + " \"Description_\" + x.Key, " . + " x.Value, " . + " new CreateFieldOptions { " . + // Index each generated dynamic field for FTS + " Indexing = FieldIndexing.Search, " . + " Storage = FieldStorage.No " . + " })) " . + "}"; + + $this->storeAllFields(FieldStorage::no()); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + $this->getConfiguration()[self::USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS] = "true"; + } + } + ``` + + + ```php + class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask + { + private const USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (product) { + return { + _: Object.keys(product.descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: 'Search', + storage: 'No' + }); + }) + }; + })" + ]); + + // Apply a storage default to every generated field + $fields = new IndexFieldOptionsArray(); + $allFieldsOptions = new IndexFieldOptions(); + $allFieldsOptions->setStorage(FieldStorage::no()); + $fields->offsetSet(DocumentsIndexingFields::ALL_FIELDS, $allFieldsOptions); + $this->setFields($fields); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + $this->getConfiguration()[self::USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS] = "true"; + } + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```php + /** @var array $results */ + $results = $session->advanced() + ->documentQuery(Product::class, Products_ByLocalizedDescription::class) + ->search("Description_English", "north wind") + ->toList(); + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + + +--- + + + +### Configuring analyzers for dynamic fields + +`CreateFieldOptions` does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field + + - -{`class Products_ByProductType extends AbstractIndexCreationTask +```php +class Products_ByLocalizedDescription extends AbstractIndexCreationTask { public function __construct() { parent::__construct(); - // Call 'CreateField' to generate the dynamic-index-fields - // The field name will be the value of document field 'ProductType' - // The field terms will be derived from document field 'PricePerUnit' - $this->map = "docs.Products.Select(p => new { " . - " _ = this.CreateField(p.productType, p.pricePerUnit) " . - "})"; + $this->map = + "from product in docs.Products " . + "select new { " . + " _ = product.descriptions.Select(x => " . + " CreateField(" . + " \"Description_\" + x.Key, " . + " x.Value, " . + " new CreateFieldOptions { " . + " Indexing = FieldIndexing.Search, " . + " Storage = FieldStorage.No " . + " })) " . + "}"; + + $this->storeAllFields(FieldStorage::no()); + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + $this->analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -{`class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask + +```php +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { public function __construct() { parent::__construct(); $this->setMaps([ - "map('Products', function (p) { + "map('Products', function (product) { return { - _: createField(p.ProductType, p.PricePerUnit, - { indexing: 'Search', storage: true, termVector: null }) + _: Object.keys(product.descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" ]); - } -} -`} - - - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Products_ByProductType::class) -// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -->whereEquals("Electronics", 23) -->toList(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -from index 'Products/ByProductType' where Electronics = 23 -`} - - - - -## Example - list - - -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. + // Apply a storage default to every generated field + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + $fields = new IndexFieldOptionsArray(); - + $allFieldsOptions = new IndexFieldOptions(); + $allFieldsOptions->setStorage(FieldStorage::no()); + $fields->offsetSet(DocumentsIndexingFields::ALL_FIELDS, $allFieldsOptions); -* **The document**: - - -{`class Product -\{ - public ?string $id = null; - public ?string $name = null; - - // For each element in this list, the VALUE of property 'PropName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - public ?AttributeList $attributes = null; - - // ... getters and setters -\} - -class Attribute -\{ - public ?string $propName = null; - public ?string $propValue = null; - - // ... getters and setters -\} - -class AttributeList extends TypedList -\{ - protected function __construct() - \{ - parent::__construct(Attribute::class); - \} -\} -`} - - + $englishFieldOptions = new IndexFieldOptions(); + $englishFieldOptions->setAnalyzer("StopAnalyzer"); + $fields->offsetSet("Description_English", $englishFieldOptions); - - -{`// Sample document content -\{ -Name": "SomeName", -Attributes": [ - \{ - "PropName": "Color", - "PropValue": "Blue" - \}, - \{ - "PropName": "Width", - "PropValue": "10" - \}, - \{ - "PropName": "Length", - "PropValue": "20" - \}, - ... - -\} -`} - + $this->setFields($fields); + } +} +``` + -* **The index**: - The below index will create a dynamic-index-field per item in the document's `Attributes` list. - New items added to the Attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's PropName **value**. - E.g., 'PropName' value `width` will be a dynamic-index-field. +#### Configure a fallback analyzer for all fields - + - -{`class Attributes_ByName extends AbstractIndexCreationTask +```php +class Products_ByLocalizedDescription extends AbstractIndexCreationTask { public function __construct() { parent::__construct(); - // Define the dynamic-index-fields by calling CreateField - // A dynamic-index-field will be generated for each item in the Attributes list + $this->map = + "from product in docs.Products " . + "select new { " . + " _ = product.descriptions.Select(x => " . + " CreateField(" . + " \"Description_\" + x.Key, " . + " x.Value, " . + " new CreateFieldOptions { " . + " Indexing = FieldIndexing.Search, " . + " Storage = FieldStorage.No " . + " })) " . + "}"; - // For each item, the field name will be the value of field 'PropName' - // The field terms will be derived from field 'PropValue' + $this->storeAllFields(FieldStorage::no()); - $this->map = - "docs.Products.Select(p => new { " . - " _ = p.attributes.Select(item => this.CreateField(item.propName, item.propValue)), " . - " Name = p.name " . - "})"; + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + $this->analyze(DocumentsIndexingFields::ALL_FIELDS, "StopAnalyzer"); } } -`} - +``` - - -{`class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask + +```php +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { public function __construct() { parent::__construct(); $this->setMaps([ - "map('Products', function (p) { + "map('Products', function (product) { return { - _: p.Attributes.map(item => createField(item.PropName, item.PropValue, - { indexing: 'Search', storage: true, termVector: null })), - Name: p.Name + _: Object.keys(product.descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" ]); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + $fields = new IndexFieldOptionsArray(); + $allFieldsOptions = new IndexFieldOptions(); + $allFieldsOptions->setStorage(FieldStorage::no()); + $allFieldsOptions->setAnalyzer("StopAnalyzer"); + $fields->offsetSet(DocumentsIndexingFields::ALL_FIELDS, $allFieldsOptions); + $this->setFields($fields); } } -`} - - - - -* **The query**: - To get all documents matching a specific attribute property use: - - - -{`/** @var array $matchingDocuments */ -$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Attributes_ByName::class) - // 'Width' is a dynamic-index-field that was indexed from the Attributes list - ->whereEquals("Width", 10) - ->toList(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the Attributes list -from index 'Attributes/ByName' where Width = 10 -`} - +``` + + + + +### Which analyzer is used for the query term? +The following applies when a `search()` query targets a **dynamic field**: -## CreateField syntax +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `$this->analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `$this->analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | - - -{`object CreateField(string name, object value); + + + + + + +#### Syntax for LINQ-index: + + +```php +object CreateField(string name, object value); object CreateField(string name, object value, bool stored, bool analyzed); object CreateField(string name, object value, CreateFieldOptions options); -`} - +``` -| Parameters | Type | Description | -|------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **name** | `string` | Name of the dynamic-index-field | -| **value** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`False` - will set `FieldStorage.NO` (default value)
`True` - will set `FieldStorate.YES` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`None` - `FieldIndexing.Default` (default value)
`False` - `FieldIndexing.Exact`
`True` - `FieldIndexing.Search` | -| **options** | `CreateFieldOptions` | Dynamic-index-field options | +#### Syntax for JavaScript-index: -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| -| **Storage** | `FieldStorage` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing` | | -| **TermVector** | `FieldTermVector` | | + +```js +createField(fieldName, fieldValue, options); // returns object +``` + + +| Parameters | Type | Description | +|----------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage::no()` (default value)
`true` - will set `FieldStorage::yes()` | +| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing::default()` (default value)
`false` - `FieldIndexing::exact()`
`true` - `FieldIndexing::search()` | +| **options** | `CreateFieldOptions` | Dynamic-index-field options | + +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| +| **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -544,17 +976,15 @@ object CreateField(string name, object value, CreateFieldOptions options); + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. - +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/docs/indexes/content/_using-dynamic-fields-python.mdx b/docs/indexes/content/_using-dynamic-fields-python.mdx index 69514a29e2..6862d18067 100644 --- a/docs/indexes/content/_using-dynamic-fields-python.mdx +++ b/docs/indexes/content/_using-dynamic-fields-python.mdx @@ -1,490 +1,886 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. - -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. - -* In this page: - - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -## Indexing documents fields KEYS - -## Example - index any field under object + - - -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + + +### Example - index every field under an object + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. * **The document**: - - -{`class Product: - def __init__(self, Id: str = None, attributes: Dict[str, object] = None): - self.Id = Id - - # The KEYS under the Attributes object will be dynamically indexed - # Fields added to this object after index creation time will also get indexed - self.attributes = attributes -`} - - - - -{`// Sample document content -\{ - "Attributes": \{ - "Color": "Red", - "Size": 42 - \} -\} -`} - - + + + ```python + class Product: + def __init__(self, Id: str = None, Attributes: Dict[str, object] = None): + self.Id = Id + + # The KEYS under the Attributes object will be dynamically indexed + # Fields added to this object after index creation time will also get indexed + self.Attributes = Attributes + ``` + + + + ```json + // Sample document content + { + "Attributes": { + "Color": "Red", + "Size": 42 + } + } + ``` + * **The index**: - The below index will index any field under the `Attributes` object from the document, + + The following index will index any field under the `Attributes` object from the document, a dynamic-index-field will be created for each such field. New fields added to the object after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAttributeKey(AbstractIndexCreationTask): - def __init__(self): - super().__init__() - self.map = ( - "from p in docs.Products select new {" - "_ = p.attributes.Select(item => CreateField(item.Key, item.Value))" - "}" - ) -`} - - - - -{`class Products_ByAttributeKey_JS(AbstractJavaScriptIndexCreationTask): - def __init__(self): - super().__init__() - self.maps = { - """ - map('Products', function (p) { - return { - _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], - { indexing: 'Search', storage: true, termVector: null })) - }; - }) - """ - } -`} - - - - + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. + + + + ```python + class Products_ByAttributeKey(AbstractIndexCreationTask): + def __init__(self): + super().__init__() + self.map = ( + "from p in docs.Products select new { " + # Call 'CreateField' to generate dynamic-index-fields + # from the Attributes object keys. + + # Using '_' is just a convention. + # Any other string can be used instead of '_' + + # The actual field name will be 'item.Key' + # The actual field terms will be derived from 'item.Value' + "_ = p.Attributes.Select(item => CreateField(item.Key, item.Value)) " + "}" + ) + ``` + + + ```python + class Products_ByAttributeKey_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (p) { + return { + // The field name will be the key + // The field terms will be derived from the corresponding value + _: Object.keys(p.Attributes) + .map(key => createField(key, p.Attributes[key])) + }; + }) + """ + ] + ``` + + * **The query**: - * You can now query the generated dynamic-index fields. - * To get all documents with some 'size' use: - - - -{`matching_documents = list( - session.query_index_type(Products_ByAttributeKey, Product) - # 'size' is a dynamic-index-field that was indexed from the attributes object - .where_equals("size", 42) -) -`} - - - - -{`// 'Size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey' where Size = 42 -`} - - - - -## Example - index any field - - + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _Size_ use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Products_ByAttributeKey, Product) + # 'Size' is a dynamic-index-field that was indexed from the Attributes object + .where_equals("Size", 42) + ) + ``` + + + ```sql + // 'Size' is a dynamic-index-field that was indexed from the Attributes object + from index 'Products/ByAttributeKey' where Size = 42 + ``` + + + + + Dynamic index fields are queried with _document_query_ or _RQL_ because the field names (e.g. `Size`, `Color`) + are generated at indexing time and are not properties on the _Product_ class. + + Use the string-based `where_equals("Size", 42)` or RQL `where Size = 42`. + + + + + + +### Example - index every field + +The following example allows you to: * Define an index on a collection **without** needing any common structure between the indexed documents. - * After index is deployed, any new field added to the document will be indexed as well. - + * After the index is deployed, any new field added to any document in the collection will be indexed as well. - + -Consider whether this is really necessary, as indexing every single field can end up costing time and disk space. +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`class Product: - def __init__(self, Id: str = None, first_name: str = None, last_name: str = None, title: str = None): - self.Id = Id - - # All KEYS in the document will be dynamically indexes - # Fields added to the document after index creation time wil also get indexed - self.first_name = first_name - self.last_name = last_name - self.title = title - # ... -`} - - - - - -{`// Sample document content - \{ + + + ```python + class Product: + def __init__( + self, + Id: str = None, + FirstName: str = None, + LastName: str = None, + Title: str = None, + ): + self.Id = Id + + # All KEYS in the document will be dynamically indexed + # Fields added to the document after index creation time will also get indexed + self.FirstName = FirstName + self.LastName = LastName + self.Title = Title + # ... + ``` + + + + ```json + // Sample document content + { "FirstName": "John", "LastName": "Doe", "Title": "Engineer", // ... -\} -`} - - + } + ``` + * **The index**: - The below index will index any field from the document, + + The following index will index any field from the document, a dynamic-index-field will be created for each field. New fields added to the document after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAnyField_JS(AbstractJavaScriptIndexCreationTask): - def __init__(self): - super().__init__() - # This will index EVERY FIELD under the top level of the document - self.maps = { - """ - map('Products', function (p) { - return { - _: Object.keys(p).map(key => createField(key, p[key], - { indexing: 'Search', storage: true, termVector: null })) - } - }) - """ - } -`} - - - + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. + + The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection. + + + + ```python + class Products_ByAnyField_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + # This will index EVERY FIELD under the top level of the document + self.maps = [ + """ + map('Products', function (p) { + return { + _: Object.keys(p).map(key => createField(key, p[key])) + }; + }) + """ + ] + ``` + + * **The query**: - * To get all documents with some 'LastName' use: - - - -{`# 'last_name' is a dynamic-index-field that was indexed from the document -matching_documents = list( - session.query_index_type(Products_ByAnyField_JS, Product).where_equals("last_name", "Doe") -) -`} - - - - -{`// 'LastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where LastName = "Doe" -`} - - - + + To get all documents with a specific _LastName_ use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Products_ByAnyField_JS, Product) + # 'LastName' is a dynamic-index-field that was indexed from the document + .where_equals("LastName", "Doe") + ) + ``` + + + ```sql + // 'LastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where LastName = "Doe" + ``` + + + + + + + + + + + +### Example - basic + + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + + +* **The document**: + + + ```python + class Product: + def __init__(self, Id: str = None, ProductType: str = None, PricePerUnit: int = None): + self.Id = Id + + # The VALUE of ProductType will be dynamically indexed + self.ProductType = ProductType + self.PricePerUnit = PricePerUnit + ``` + + + + ```json + // Sample document content + { + "ProductType": "Electronics", + "PricePerUnit": 23 + } + ``` + + +* **The index**: + + The following index will index the **value** of document field 'ProductType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + + ```python + class Products_ByProductType(AbstractIndexCreationTask): + def __init__(self): + super().__init__() + self.map = ( + "from p in docs.Products select new { " + # Call 'CreateField' to generate the dynamic-index-fields + # The field name will be the value of document field 'ProductType' + # The field terms will be derived from document field 'PricePerUnit' + "_ = CreateField(p.ProductType, p.PricePerUnit) " + "}" + ) + ``` + + + ```python + class Products_ByProductType_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (p) { + return { + // The field name will be the value of document field 'ProductType' + // The field terms will be derived from document field 'PricePerUnit' + _: createField(p.ProductType, p.PricePerUnit) + }; + }) + """ + ] + ``` + + + +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Products_ByProductType, Product) + # 'Electronics' is the dynamic-index-field that was indexed + # from document field 'ProductType' + .where_equals("Electronics", 23) + ) + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'ProductType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + + +### Example - list + + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. + + +* **The document**: + + + ```python + class Attribute: + def __init__(self, PropName: str = None, PropValue: str = None): + self.PropName = PropName + self.PropValue = PropValue + + + class Product: + def __init__(self, Id: str = None, Name: str = None, Attributes: List[Attribute] = None): + self.Id = Id + self.Name = Name + + # For each element in this list, + # the VALUE of property 'PropName' will be dynamically indexed. + # e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + self.Attributes = Attributes + ``` + + + + ```json + // Sample document content + { + "Name": "SomeName", + "Attributes": [ + { + "PropName": "Color", + "PropValue": "Blue" + }, + { + "PropName": "Width", + "PropValue": "10" + }, + { + "PropName": "Length", + "PropValue": "20" + }, + // ... + ] + } + ``` + + +* **The index**: + + The following index will create a dynamic-index-field per item in the document's `Attributes` list. + New items added to the Attributes list after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the item's PropName **value**. + E.g., 'PropName' value `Width` will be a dynamic-index-field. + + + + ```python + class Attributes_ByName(AbstractIndexCreationTask): + def __init__(self): + super().__init__() + self.map = ( + "from a in docs.Products select new { " + # Define the dynamic-index-fields by calling 'CreateField' + # A dynamic-index-field will be generated for each item in 'Attributes' + + # For each item, the field name will be the value of field 'PropName' + # The field terms will be derived from field 'PropValue' + "_ = a.Attributes.Select(item => CreateField(item.PropName, item.PropValue)), " + + # A regular index field can be defined as well: + "Name = a.Name " + "}" + ) + ``` + + + ```python + class Attributes_ByName_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (p) { + return { + // For each item, + // the field name will be the value of field 'PropName' + // The field terms will be derived from field 'PropValue' + _: p.Attributes.map(item => createField(item.PropName, item.PropValue)), + + // A regular index field can be defined as well: + Name: p.Name + }; + }) + """ + ] + ``` + + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Attributes_ByName, Product) + # 'Width' is a dynamic-index-field that was indexed from the Attributes list + .where_equals("Width", 10) + ) + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the Attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `FieldIndexing.Search` in `CreateFieldOptions`. +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). -## Indexing documents fields VALUES +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. -## Example - basic + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. - + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```python + class Product: + def __init__(self, Id: str = None, Descriptions: Dict[str, str] = None): + self.Id = Id + self.Descriptions = Descriptions + ``` + + + + ```json + // Sample document content + { + "Descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + -* **The document**: - - -{`class Product: - def __init__(self, Id: str = None, product_type: str = None, price_per_unit: float = None): - self.Id = Id - - # The VALUE of ProductType will be dynamically indexed - self.product_type = product_type - self.price_per_unit = price_per_unit -`} - - +* **The index**: - - -{`// Sample document content -\{ - "ProductType": "Electronics", - "PricePerUnit": 23 -\} -`} - - + The following index creates one dynamic field for each entry in the `Descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. -* **The index**: - The below index will index the **value** of document field 'ProductType'. - - This value will be the dynamic-index-field name on which you can query. - E.g., Field value `Electronics` will be the dynamic-index-field. + Each generated dynamic field is indexed for **full-text search**. + + + + ```python + class Products_ByLocalizedDescription(AbstractIndexCreationTask): + USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = ( + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery" + ) + + def __init__(self): + super().__init__() + self.map = ( + "from product in docs.Products select new { " + " _ = product.Descriptions.Select(x => " + " CreateField(" + " \"Description_\" + x.Key, " + " x.Value, " + # Index each generated dynamic field for FTS + " new CreateFieldOptions { " + " Indexing = FieldIndexing.Search, " + " Storage = FieldStorage.No " + " })) " + "}" + ) + + self._store_all_fields(FieldStorage.NO) + + # Set this configuration key to true so that search() queries on dynamic fields + # will use the 'Default Search Analyzer' when the generated field name + # has no explicit analyzer configuration. + self.configuration = { + Products_ByLocalizedDescription.USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS: "true" + } + ``` + + + ```python + class Products_ByLocalizedDescription_JS(AbstractJavaScriptIndexCreationTask): + USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = ( + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery" + ) + + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (product) { + return { + _: Object.keys(product.Descriptions).map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: 'Search', + storage: 'No' + }); + }) + }; + }) + """ + ] + + # Apply a storage default to every generated field + self.fields = { + constants.Documents.Indexing.Fields.ALL_FIELDS: + IndexFieldOptions(storage=FieldStorage.NO) + } + + # Set this configuration key to true so that search() queries on dynamic fields + # will use the 'Default Search Analyzer' when the generated field name + # has no explicit analyzer configuration. + self.configuration = { + Products_ByLocalizedDescription_JS.USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS: "true" + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```python + results = list( + session.advanced + .document_query_from_index_type(Products_ByLocalizedDescription, Product) + .search("Description_English", "north wind") + ) + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + + +--- + + + +### Configuring analyzers for dynamic fields + +`CreateFieldOptions` does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field - -{`class Products_ByProductType(AbstractIndexCreationTask): +```python +class Products_ByLocalizedDescription(AbstractIndexCreationTask): def __init__(self): super().__init__() + self.map = ( + "from product in docs.Products select new { " + " _ = product.Descriptions.Select(x => " + " CreateField(" + " \"Description_\" + x.Key, " + " x.Value, " + " new CreateFieldOptions { " + " Indexing = FieldIndexing.Search, " + " Storage = FieldStorage.No " + " })) " + "}" + ) - # Call 'CreateField' to generate the dynamic-index-fields - # The field name will be the value of document field 'product_type' - # The field terms will be derived from document field 'price_per_unit' - self.map = "from p in docs.Products select new { _ = CreateField(p.product_type, p.price_per_unit)}" -`} - + self._store_all_fields(FieldStorage.NO) + + # Configure an analyzer for a known generated dynamic field name + # ============================================================== + self._analyze("Description_English", "StopAnalyzer") +``` - - -{`class Products_ByProductType_JS(AbstractJavaScriptIndexCreationTask): + +```python +class Products_ByLocalizedDescription_JS(AbstractJavaScriptIndexCreationTask): def __init__(self): super().__init__() - self.maps = { + self.maps = [ """ - map('Products', function (p) { + map('Products', function (product) { return { - _: createField(p.product_type, p.price_per_unit, - { indexing: 'Search', storage: true, termVector: null }) + _: Object.keys(product.Descriptions).map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; }) """ + ] + + # Configure an analyzer for a known generated dynamic field name + # ============================================================== + self.fields = { + constants.Documents.Indexing.Fields.ALL_FIELDS: + IndexFieldOptions(storage=FieldStorage.NO), + "Description_English": + IndexFieldOptions(analyzer="StopAnalyzer") } -`} - - - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`# 'electronics' is the dynamic-index-field that was indexed from the document 'product_type' -matching_documents = list( - session.advanced.document_query_from_index_type(Products_ByProductType, Product).where_equals( - "electronics", 23 - ) -) -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -from index 'Products/ByProductType' where Electronics = 23 -`} - +``` -## Example - list - - - -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. - - - -* **The document**: - - -{`class Attribute: - def __init__(self, prop_name: str = None, prop_value: str = None): - self.prop_name = prop_name - self.prop_value = prop_value - - -class Product: - def __init__(self, Id: str = None, name: str = None, attributes: List[Attribute] = None): - self.Id = Id - self.name = name - # For each element in this list, the VALUE of property 'prop_name' will be dynamically indexed - # e.g. color, width, length (in ex. below) will become dynamic-index-field - self.attributes = attributes -`} - - - - - -{`// Sample document content -\{ -Name": "SomeName", -Attributes": [ - \{ - "PropName": "Color", - "PropValue": "Blue" - \}, - \{ - "PropName": "Width", - "PropValue": "10" - \}, - \{ - "PropName": "Length", - "PropValue": "20" - \}, - ... - -\} -`} - - - -* **The index**: - The below index will create a dynamic-index-field per item in the document's `Attributes` list. - New items added to the Attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's PropName **value**. - E.g., 'PropName' value `width` will be a dynamic-index-field. +#### Configure a fallback analyzer for all fields - -{`class Attributes_ByName(AbstractIndexCreationTask): +```python +class Products_ByLocalizedDescription(AbstractIndexCreationTask): def __init__(self): super().__init__() self.map = ( - "from a in docs.Products select new " - "{ _ = a.attributes.Select( item => CreateField(item.prop_name, item.prop_value)), name = a.name " + "from product in docs.Products select new { " + " _ = product.Descriptions.Select(x => " + " CreateField(" + " \"Description_\" + x.Key, " + " x.Value, " + " new CreateFieldOptions { " + " Indexing = FieldIndexing.Search, " + " Storage = FieldStorage.No " + " })) " "}" ) -`} - + + self._store_all_fields(FieldStorage.NO) + + # Use this as a fallback for fields that do not have + # their own explicit analyzer configuration + # ================================================== + self._analyze(constants.Documents.Indexing.Fields.ALL_FIELDS, "StopAnalyzer") +``` - - -{`class Attributes_ByName_JS(AbstractJavaScriptIndexCreationTask): + +```python +class Products_ByLocalizedDescription_JS(AbstractJavaScriptIndexCreationTask): def __init__(self): super().__init__() - self.maps = { + self.maps = [ """ - map('Products', function (p) { + map('Products', function (product) { return { - _: p.Attributes.map(item => createField(item.PropName, item.PropValue, - { indexing: 'Search', storage: true, termVector: null })), - Name: p.Name + _: Object.keys(product.Descriptions).map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; }) """ + ] + + # Use this as a fallback for fields that do not have + # their own explicit analyzer configuration + # ================================================== + self.fields = { + constants.Documents.Indexing.Fields.ALL_FIELDS: + IndexFieldOptions(storage=FieldStorage.NO, analyzer="StopAnalyzer") } -`} - - - - -* **The query**: - To get all documents matching a specific attribute property use: - - - -{`matching_documents = list( - session.advanced.document_query_from_index_type(Attributes_ByName, Product).where_equals( - "width", 10 - ) -) -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the Attributes list -from index 'Attributes/ByName' where Width = 10 -`} - +``` + + + + +### Which analyzer is used for the query term? + +The following applies when a `search()` query targets a **dynamic field**: + +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `self._analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `self._analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | + + + -## CreateField syntax + #### Syntax for Index: - - -{`object CreateField(string name, object value); + +```python +CreateField(name, value) -object CreateField(string name, object value, bool stored, bool analyzed); +CreateField(name, value, stored, analyzed) -object CreateField(string name, object value, CreateFieldOptions options); -`} - +CreateField(name, value, CreateFieldOptions) +``` #### Syntax for JavaScript-index: - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` -| Parameters | Type | Description | -|------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **name** | `string` | Name of the dynamic-index-field | -| **value** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`False` - will set `FieldStorage.NO` (default value)
`True` - will set `FieldStorate.YES` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`None` - `FieldIndexing.Default` (default value)
`False` - `FieldIndexing.Exact`
`True` - `FieldIndexing.Search` | -| **options** | `CreateFieldOptions` | Dynamic-index-field options | +| Parameters | Type | Description | +|----------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorage.Yes` | +| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | +| **options** | `CreateFieldOptions` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| | **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -496,17 +892,15 @@ object CreateField(string name, object value, CreateFieldOptions options); + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/docs/indexes/querying/content/_searching-csharp.mdx b/docs/indexes/querying/content/_searching-csharp.mdx index d20ef5a65e..f0be0e6ffd 100644 --- a/docs/indexes/querying/content/_searching-csharp.mdx +++ b/docs/indexes/querying/content/_searching-csharp.mdx @@ -16,11 +16,13 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) * [Searching with wildcards](../../../indexes/querying/searching.mdx#searching-with-wildcards) * [When using RavenStandardAnalyzer or StandardAnalyzer or NGramAnalyzer](../../../indexes/querying/searching.mdx#when-usingoror) @@ -28,6 +30,7 @@ import CodeBlock from '@theme/CodeBlock'; * [When using the Exact analyzer](../../../indexes/querying/searching.mdx#when-using-the-exact-analyzer) + ## Indexing single field for FTS #### The index: @@ -262,7 +265,7 @@ where (search(EmployeeData, "Manager") or search(EmployeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method in the _Map_ function to extract all property values and index them in a single searchable field. * This approach makes the index robust to changes in the document schema. @@ -377,7 +380,14 @@ where search(AllValues, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). ## Boosting search results diff --git a/docs/indexes/querying/content/_searching-java.mdx b/docs/indexes/querying/content/_searching-java.mdx index 7d85246ced..4accf3f512 100644 --- a/docs/indexes/querying/content/_searching-java.mdx +++ b/docs/indexes/querying/content/_searching-java.mdx @@ -388,4 +388,11 @@ where search(allValues, "tofu") +## Indexing dynamic fields for FTS +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). \ No newline at end of file diff --git a/docs/indexes/querying/content/_searching-nodejs.mdx b/docs/indexes/querying/content/_searching-nodejs.mdx index 4ca66b5877..cb73bfd7c0 100644 --- a/docs/indexes/querying/content/_searching-nodejs.mdx +++ b/docs/indexes/querying/content/_searching-nodejs.mdx @@ -5,7 +5,7 @@ import CodeBlock from '@theme/CodeBlock'; -* Prior to reading this article, please refer to [full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) +* Prior to reading this article, please refer to [Full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) to learn about the `search` method. * **All capabilities** provided by `search` with a dynamic query can also be used when querying a static-index. @@ -17,11 +17,13 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) * [Searching with wildcards](../../../indexes/querying/searching.mdx#searching-with-wildcards) * [When using RavenStandardAnalyzer or StandardAnalyzer or NGramAnalyzer](../../../indexes/querying/searching.mdx#when-usingoror) @@ -29,6 +31,7 @@ import CodeBlock from '@theme/CodeBlock'; * [When using the Exact analyzer](../../../indexes/querying/searching.mdx#when-using-the-exact-analyzer) + ## Indexing single field for FTS #### The index: @@ -163,7 +166,7 @@ where (search(employeeData, "Manager") or search(employeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, which is available when using **a C# LINQ string** that is assigned to the `map` property in the Node.js index class, as shown in the example below. @@ -229,6 +232,14 @@ where search(AllValues, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). ## Boosting search results diff --git a/docs/indexes/querying/content/_searching-php.mdx b/docs/indexes/querying/content/_searching-php.mdx index 55261ce19f..510a6f8752 100644 --- a/docs/indexes/querying/content/_searching-php.mdx +++ b/docs/indexes/querying/content/_searching-php.mdx @@ -5,7 +5,7 @@ import CodeBlock from '@theme/CodeBlock'; -* Prior to reading this article, please refer to [full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) +* Prior to reading this article, please refer to [Full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) to learn about the `search` method. * **All capabilities** provided by `search` with a dynamic query can also be used when querying a static-index. @@ -17,14 +17,17 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) + ## Indexing single field for FTS #### The index: @@ -253,7 +256,7 @@ where (search(EmployeeData, "Manager") or search(EmployeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, which is available in the **C# LINQ string** that is assigned to the `map` property in the PHP index class, as shown in the example below. @@ -332,6 +335,15 @@ where search(allValues, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). + ## Boosting search results diff --git a/docs/indexes/querying/content/_searching-python.mdx b/docs/indexes/querying/content/_searching-python.mdx index f06c29d94f..88c6aa452f 100644 --- a/docs/indexes/querying/content/_searching-python.mdx +++ b/docs/indexes/querying/content/_searching-python.mdx @@ -5,7 +5,7 @@ import CodeBlock from '@theme/CodeBlock'; -* Prior to reading this article, please refer to [full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) +* Prior to reading this article, please refer to [Full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) to learn about the `search` method. * **All capabilities** provided by `search` with a dynamic query can also be used when querying a static-index. @@ -17,14 +17,17 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) + ## Indexing single field for FTS #### The index: @@ -167,7 +170,7 @@ where (search(EmployeeData, "Manager") or search(EmployeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, which is available in the **C# LINQ string** that is assigned to the `map` property in the Python index class, as shown in the example below. @@ -238,6 +241,14 @@ where search(all_values, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). ## Boosting search results diff --git a/docs/indexes/using-dynamic-fields.mdx b/docs/indexes/using-dynamic-fields.mdx index 19b5239557..e465ba5431 100644 --- a/docs/indexes/using-dynamic-fields.mdx +++ b/docs/indexes/using-dynamic-fields.mdx @@ -1,6 +1,6 @@ --- -title: "Indexes: Dynamic Index Fields" -sidebar_label: Dynamic Fields +title: "Dynamic Index Fields" +sidebar_label: "Dynamic Index Fields" description: "Index document fields dynamically in RavenDB when field names are unknown at index definition time, using the CreateField method." sidebar_position: 27 supported_languages: ["csharp", "java", "python", "php", "nodejs"] @@ -32,7 +32,6 @@ import UsingDynamicFieldsPython from './content/_using-dynamic-fields-python.mdx import UsingDynamicFieldsPhp from './content/_using-dynamic-fields-php.mdx'; import UsingDynamicFieldsNodejs from './content/_using-dynamic-fields-nodejs.mdx'; - diff --git a/docs/server/configuration/indexing-configuration.mdx b/docs/server/configuration/indexing-configuration.mdx index 132bf987f5..0abdfc4f72 100644 --- a/docs/server/configuration/indexing-configuration.mdx +++ b/docs/server/configuration/indexing-configuration.mdx @@ -104,6 +104,7 @@ import LanguageContent from "@site/src/components/LanguageContent"; [Indexing.OrderByTicksAutomaticallyWhenDatesAreInvolved](../../server/configuration/indexing-configuration.mdx#indexingorderbyticksautomaticallywhendatesareinvolved) [Indexing.QueryClauseCache.Disabled](../../server/configuration/indexing-configuration.mdx#indexingqueryclausecachedisabled) [Indexing.QueryClauseCache.RepeatedQueriesTimeFrameInSec](../../server/configuration/indexing-configuration.mdx#indexingqueryclausecacherepeatedqueriestimeframeinsec) + [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) [Indexing.ScratchSpaceLimitInMb](../../server/configuration/indexing-configuration.mdx#indexingscratchspacelimitinmb) [Indexing.Static.SearchEngineType](../../server/configuration/indexing-configuration.mdx#indexingstaticsearchenginetype) [Indexing.Throttling.TimeIntervalInMs](../../server/configuration/indexing-configuration.mdx#indexingthrottlingtimeintervalinms) @@ -1009,6 +1010,33 @@ Queries that repeat within this time frame will be considered worth caching. +## Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery + +* This setting controls which analyzer is used for the **query term** when a **`search()` query** targets a [Dynamic index field](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + that was indexed for full-text search, and no analyzer is explicitly configured for that field in the index definition. + + * `false` (default): + The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) is used to analyze the **query term**. + This analyzer can be configured via the [Indexing.Analyzers.Default](../../server/configuration/indexing-configuration.mdx#indexinganalyzersdefault) configuration key. + + * `true`: + The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) is used to analyze the **query term**. + This analyzer can be configured via the [Indexing.Analyzers.Search.Default](../../server/configuration/indexing-configuration.mdx#indexinganalyzerssearchdefault) configuration key. + +* See the table in [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + for the complete analyzer-selection rules based on this configuration key and the analyzer configured for the dynamic index field. + +* This configuration applies to both the Lucene and Corax search engines. + The default is `false` to preserve backward compatibility. + +--- + +- **Type**: `bool` +- **Default**: `false` +- **Scope**: Server-wide, or per database, or per index + + + ## Indexing.ScratchSpaceLimitInMb * Amount of scratch space in megabytes that we allow to use for the index storage. diff --git a/versioned_docs/version-6.2/indexes/assets/dynamic-index-fields-1.png b/versioned_docs/version-6.2/indexes/assets/dynamic-index-fields-1.png index b13d9076f8..a20adda0ec 100644 Binary files a/versioned_docs/version-6.2/indexes/assets/dynamic-index-fields-1.png and b/versioned_docs/version-6.2/indexes/assets/dynamic-index-fields-1.png differ diff --git a/versioned_docs/version-6.2/indexes/assets/dynamic-index-fields-2.png b/versioned_docs/version-6.2/indexes/assets/dynamic-index-fields-2.png index dd2b4082ea..c5b1bc14bc 100644 Binary files a/versioned_docs/version-6.2/indexes/assets/dynamic-index-fields-2.png and b/versioned_docs/version-6.2/indexes/assets/dynamic-index-fields-2.png differ diff --git a/versioned_docs/version-6.2/indexes/content/_using-analyzers-csharp.mdx b/versioned_docs/version-6.2/indexes/content/_using-analyzers-csharp.mdx index 6022cfd145..2e824fea4b 100644 --- a/versioned_docs/version-6.2/indexes/content/_using-analyzers-csharp.mdx +++ b/versioned_docs/version-6.2/indexes/content/_using-analyzers-csharp.mdx @@ -24,7 +24,7 @@ import CodeBlock from '@theme/CodeBlock'; * [Understanding the role of analyzers](../../indexes/using-analyzers.mdx#understanding-the-role-of-analyzers) * [Analyzers available in RavenDB](../../indexes/using-analyzers.mdx#analyzers-available-in-ravendb) * [Setting analyzer for index-field](../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field) - * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendb) + * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendbs-default-analyzers) * [Disabling indexing for index-field](../../indexes/using-analyzers.mdx#disabling-indexing-for-index-field) * [Creating custom analyzers](../../indexes/using-analyzers.mdx#creating-custom-analyzers) * [Viewing the indexed terms](../../indexes/using-analyzers.mdx#viewing-the-indexed-terms) diff --git a/versioned_docs/version-6.2/indexes/content/_using-analyzers-nodejs.mdx b/versioned_docs/version-6.2/indexes/content/_using-analyzers-nodejs.mdx index 6afd3b6202..fae2cdd2b9 100644 --- a/versioned_docs/version-6.2/indexes/content/_using-analyzers-nodejs.mdx +++ b/versioned_docs/version-6.2/indexes/content/_using-analyzers-nodejs.mdx @@ -24,7 +24,7 @@ import CodeBlock from '@theme/CodeBlock'; * [Understanding the role of analyzers](../../indexes/using-analyzers.mdx#understanding-the-role-of-analyzers) * [Analyzers available in RavenDB](../../indexes/using-analyzers.mdx#analyzers-available-in-ravendb) * [Setting analyzer for index-field](../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field) - * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendb) + * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendbs-default-analyzers) * [Disabling indexing for index-field](../../indexes/using-analyzers.mdx#disabling-indexing-for-index-field) * [Creating custom analyzers](../../indexes/using-analyzers.mdx#creating-custom-analyzers) * [Viewing the indexed terms](../../indexes/using-analyzers.mdx#viewing-the-indexed-terms) diff --git a/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-csharp.mdx b/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-csharp.mdx index b4c4f518df..5bf052d482 100644 --- a/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-csharp.mdx +++ b/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-csharp.mdx @@ -2,533 +2,923 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. - -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. - -* In this page: - - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -## Indexing documents fields KEYS - -## Example - index any field under object - - + -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + + +### Example - index every field under an object + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. * **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - - // The KEYS under the Attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - public Dictionary Attributes \{ get; set; \} -\} -`} - - - - -{`// Sample document content -\{ - "Attributes": \{ - "Color": "Red", - "Size": 42 - \} -\} -`} - - + + + ```csharp + public class Product + { + public string Id { get; set; } + + // The KEYS under the Attributes object will be dynamically indexed + // Fields added to this object after index creation time will also get indexed + public Dictionary Attributes { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "Attributes": { + "Color": "Red", + "Size": 42 + } + } + ``` + * **The index**: - The below index will index any field under the `Attributes` object from the document, + + The following index will index any field under the `Attributes` object from the document, a dynamic-index-field will be created for each such field. New fields added to the object after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. - - - -{`public class Products_ByAttributeKey : AbstractIndexCreationTask -{ - public Products_ByAttributeKey() + + + ```csharp + public class Products_ByAttributeKey : AbstractIndexCreationTask { - Map = products => from p in products - select new - { - // Call 'CreateField' to generate dynamic-index-fields from the Attributes object keys - // Using '_' is just a convention. Any other string can be used instead of '_' - - // The actual field name will be item.Key - // The actual field terms will be derived from item.Value - _ = p.Attributes.Select(item => CreateField(item.Key, item.Value)) - }; + public Products_ByAttributeKey() + { + Map = products => from p in products + select new + { + // Call 'CreateField' to generate dynamic-index-fields + // from the Attributes object keys. + + // Using '_' is just a convention. + // Any other string can be used instead of '_' + + // The actual field name will be 'item.Key' + // The actual field terms will be derived from 'item.Value' + _ = p.Attributes.Select(item => CreateField(item.Key, item.Value)) + }; + } } -} -`} - - - - -{`public class Products_ByAttributeKey_JS : AbstractJavaScriptIndexCreationTask -{ - public Products_ByAttributeKey_JS() + ``` + + + ```csharp + public class Products_ByAttributeKey_JS : AbstractJavaScriptIndexCreationTask { - Maps = new HashSet + public Products_ByAttributeKey_JS() { - @"map('Products', function (p) { - return { - _: Object.keys(p.Attributes).map(key => createField(key, p.Attributes[key], - { indexing: 'Search', storage: true, termVector: null })) - }; - })" - }; + Maps = new HashSet + { + @"map('Products', function (p) { + return { + // The field name will be the key + // The field terms will be derived from the corresponding value + _: Object.keys(p.Attributes) + .map(key => createField(key, p.Attributes[key])) + }; + })" + }; + } } -} -`} - - - - + ``` + + * **The query**: - * You can now query the generated dynamic-index fields. - The `_` property is Not queryable but used only in the index definition syntax. - * To get all documents with some 'Size' use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'Size' is a dynamic-index-field that was indexed from the Attributes object - .WhereEquals("Size", 42) - .ToList(); -`} - - - - -{`// 'Size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey' where Size = 42 -`} - - - + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _Size_ use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'Size' is a dynamic-index-field that was indexed from the Attributes object + .WhereEquals("Size", 42) + .ToList(); + ``` + + + ```sql + // 'Size' is a dynamic-index-field that was indexed from the Attributes object + from index 'Products/ByAttributeKey' where Size = 42 + ``` + + + + + Dynamic index fields are queried with _DocumentQuery_ or _RQL_ because the field names (e.g. `Size`, `Color`) + are generated at indexing time and are not properties on the _Product_ class. + + A strongly typed LINQ query such as `Query().Where(p => p.Size == 42)` would not compile. + Use the string-based `WhereEquals("Size", 42)` or RQL `where Size = 42` instead. + -## Example - index any field + - + + +### Example - index every field + +The following example allows you to: * Define an index on a collection **without** needing any common structure between the indexed documents. - * After index is deployed, any new field added to the document will be indexed as well. - + * After the index is deployed, any new field added to any document in the collection will be indexed as well. - + -Consider whether this is really necessary, as indexing every single field can end up costing time and disk space. +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - public string FirstName \{ get; set; \} - public string LastName \{ get; set; \} - public string Title \{ get; set; \} - // ... -\} -`} - - - - - -{`// Sample document content - \{ - "FirstName": "John", - "LastName": "Doe", - "Title": "Engineer", + + + ```csharp + public class Product + { + public string Id { get; set; } + + // All KEYS in the document will be dynamically indexed + // Fields added to the document after index creation time will also get indexed + public string FirstName { get; set; } + public string LastName { get; set; } + public string Title { get; set; } // ... -\} -`} - - + } + ``` + + + + ```json + // Sample document content + { + "FirstName": "John", + "LastName": "Doe", + "Title": "Engineer", + // ... + } + ``` + * **The index**: - The below index will index any field from the document, + + The following index will index any field from the document, a dynamic-index-field will be created for each field. New fields added to the document after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. + + The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection. - - - -{`public class Products_ByAnyField_JS : AbstractJavaScriptIndexCreationTask -{ - public Products_ByAnyField_JS() + + + ```csharp + public class Products_ByAnyField_JS : AbstractJavaScriptIndexCreationTask { - // This will index EVERY FIELD under the top level of the document - Maps = new HashSet + public Products_ByAnyField_JS() { - @"map('Products', function (p) { - return { - _: Object.keys(p).map(key => createField(key, p[key], - { indexing: 'Search', storage: true, termVector: null })) - } - })" - }; + // This will index EVERY FIELD under the top level of the document + Maps = new HashSet + { + @"map('Products', function (p) { + return { + _: Object.keys(p).map(key => createField(key, p[key])) + }; + })" + }; + } } -} -`} - - - + ``` + + * **The query**: - * To get all documents with some 'LastName' use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'LastName' is a dynamic-index-field that was indexed from the document - .WhereEquals("LastName", "Doe") - .ToList(); -`} - - - - -{`// 'LastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where LastName = "Doe" -`} - - - + + To get all documents with a specific _LastName_ use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'LastName' is a dynamic-index-field that was indexed from the document + .WhereEquals("LastName", "Doe") + .ToList(); + ``` + + + ```sql + // 'LastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where LastName = "Doe" + ``` + + + + + + + + + +### Example - basic -## Indexing documents fields VALUES + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + -## Example - basic +* **The document**: + + + ```csharp + public class Product + { + public string Id { get; set; } + + // The VALUE of ProductType will be dynamically indexed + public string ProductType { get; set; } + public int PricePerUnit { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "ProductType": "Electronics", + "PricePerUnit": 23 + } + ``` + - +* **The index**: + + The following index will index the **value** of document field 'ProductType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + + ```csharp + public class Products_ByProductType : AbstractIndexCreationTask + { + public Products_ByProductType() + { + Map = products => from p in products + select new + { + // Call 'CreateField' to generate the dynamic-index-fields + // The field name will be the value of document field 'ProductType' + // The field terms will be derived from document field 'PricePerUnit' + _ = CreateField(p.ProductType, p.PricePerUnit) + }; + } + } + ``` + + + ```csharp + public class Products_ByProductType_JS : AbstractJavaScriptIndexCreationTask + { + public Products_ByProductType_JS() + { + Maps = new HashSet + { + @"map('Products', function (p) { + return { + // The field name will be the value of document field 'ProductType' + // The field terms will be derived from document field 'PricePerUnit' + _: createField(p.ProductType, p.PricePerUnit) + }; + })" + }; + } + } + ``` + + + +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'ProductType' + .WhereEquals("Electronics", 23) + .ToList(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'ProductType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. -* For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---index-a-list-of-properties) below. +### Example - list + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. * **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - - // The VALUE of ProductType will be dynamically indexed - public string ProductType \{ get; set; \} - public int PricePerUnit \{ get; set; \} -\} -`} - - - - - -{`// Sample document content -\{ - "ProductType": "Electronics", - "PricePerUnit": 23 -\} -`} - - + + + ```csharp + public class Product + { + public string Id { get; set; } + public string Name { get; set; } + + // For each element in this list, + // the VALUE of property 'PropName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + public List Attributes { get; set; } + } + + public class Attribute + { + public string PropName { get; set; } + public string PropValue { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "Name": "SomeName", + "Attributes": [ + { + "PropName": "Color", + "PropValue": "Blue" + }, + { + "PropName": "Width", + "PropValue": "10" + }, + { + "PropName": "Length", + "PropValue": "20" + }, + ... + ] + } + ``` + * **The index**: - The below index will index the **value** of document field 'ProductType'. + + The following index will create a dynamic-index-field per item in the document's `Attributes` list. + New items added to the Attributes list after index creation time will be dynamically indexed as well. - This value will be the dynamic-index-field name on which you can query. - E.g., Field value `Electronics` will be the dynamic-index-field. + The actual dynamic-index-field name on which you can query will be the item's PropName **value**. + E.g., 'PropName' value `Width` will be a dynamic-index-field. + + + + ```csharp + public class Attributes_ByName : AbstractIndexCreationTask + { + public Attributes_ByName() + { + Map = products => from a in products + select new + { + // Define the dynamic-index-fields by calling 'CreateField' + // A dynamic-index-field will be generated for each item in 'Attributes' + + // For each item, the field name will be the value of field 'PropName' + // The field terms will be derived from field 'PropValue' + _ = a.Attributes.Select(item => + CreateField(item.PropName, item.PropValue)), + + // A regular index field can be defined as well: + Name = a.Name + }; + } + } + ``` + + + ```csharp + public class Attributes_ByName_JS : AbstractJavaScriptIndexCreationTask + { + public Attributes_ByName_JS() + { + Maps = new HashSet + { + @"map('Products', function (p) { + return { + // For each item, + // the field name will be the value of field 'PropName' + // The field terms will be derived from field 'PropValue' + _: p.Attributes.map(item => createField( + item.PropName, item.PropValue)), + + // A regular index field can be defined as well: + Name: p.Name + }; + })" + }; + } + } + ``` + + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'Width' is a dynamic-index-field that was indexed from the Attributes list + .WhereEquals("Width", 10) + .ToList(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the Attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `FieldIndexing.Search` in `CreateFieldOptions`. + +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `AllFields` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `AllFields` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. + + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. + + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + + + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```csharp + public class Product + { + public string Id { get; set; } + + public Dictionary Descriptions { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "Descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `Descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. + + Each generated dynamic field is indexed for **full-text search**. + + + + ```csharp + public class Products_ByLocalizedDescription : AbstractIndexCreationTask + { + private const string UseSearchAnalyzerForDynamicFields = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription() + { + Map = products => from product in products + select new + { + _ = product.Descriptions.Select(x => + CreateField( + "Description_" + x.Key, + x.Value, + new CreateFieldOptions + { + // Index each generated dynamic field for FTS + Indexing = FieldIndexing.Search, + Storage = FieldStorage.No + })) + }; + + StoreAllFields(FieldStorage.No); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + Configuration[UseSearchAnalyzerForDynamicFields] = "true"; + } + } + ``` + + + ```csharp + public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask + { + private const string UseSearchAnalyzerForDynamicFields = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription_JS() + { + Maps = new HashSet + { + @"map('Products', function (product) { + return { + _: Object.keys(product.Descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: 'Search', + storage: 'No' + }); + }) + }; + })" + }; + + Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions + { + Storage = FieldStorage.No + }; + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + Configuration[UseSearchAnalyzerForDynamicFields] = "true"; + } + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```csharp + List results = session.Advanced + .DocumentQuery() + .Search("Description_English", "north wind") + .ToList(); + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + + +--- + + + +### Configuring analyzers for dynamic fields - - - -{`public class Products_ByProductType : AbstractIndexCreationTask +`CreateFieldOptions` does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `AllFields` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `AllFields` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field + + + +```csharp +public class Products_ByLocalizedDescription : AbstractIndexCreationTask { - public Products_ByProductType() + public Products_ByLocalizedDescription() { - Map = products => from p in products + Map = products => from product in products select new { - // Call 'CreateField' to generate the dynamic-index-fields - // The field name will be the value of document field 'ProductType' - // The field terms will be derived from document field 'PricePerUnit' - _ = CreateField(p.ProductType, p.PricePerUnit) + _ = product.Descriptions.Select(x => + CreateField( + "Description_" + x.Key, + x.Value, + new CreateFieldOptions + { + Indexing = FieldIndexing.Search, + Storage = FieldStorage.No + })) }; + + StoreAllFields(FieldStorage.No); + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + Analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -{`public class Products_ByProductType_JS : AbstractJavaScriptIndexCreationTask + +```csharp +public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask { - public Products_ByProductType_JS() + public Products_ByLocalizedDescription_JS() { Maps = new HashSet { - @"map('Products', function (p) { + @"map('Products', function (product) { return { - _: createField(p.ProductType, p.PricePerUnit, - { indexing: 'Search', storage: true, termVector: null }) + _: Object.keys(product.Descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" }; + + Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions + { + Storage = FieldStorage.No + }; + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + Fields["Description_English"] = new IndexFieldOptions + { + Analyzer = "StopAnalyzer" + }; } } -`} - - - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' - .WhereEquals("Electronics", 23) - .ToList(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -from index 'Products/ByProductType' where Electronics = 23 -`} - +``` -## Example - list - - +#### Configure a fallback analyzer for all fields -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. - - - -* **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - public string Name \{ get; set; \} - - // For each element in this list, the VALUE of property 'PropName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - public List Attributes \{ get; set; \} -\} - -public class Attribute -\{ - public string PropName \{ get; set; \} - public string PropValue \{ get; set; \} -\} -`} - - - - - -{`// Sample document content -\{ -Name": "SomeName", -Attributes": [ - \{ - "PropName": "Color", - "PropValue": "Blue" - \}, - \{ - "PropName": "Width", - "PropValue": "10" - \}, - \{ - "PropName": "Length", - "PropValue": "20" - \}, - ... - -\} -`} - - - -* **The index**: - The below index will create a dynamic-index-field per item in the document's `Attributes` list. - New items added to the Attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's PropName **value**. - E.g., 'PropName' value `Width` will be a dynamic-index-field. - - - - -{`public class Attributes_ByName : AbstractIndexCreationTask + + +```csharp +public class Products_ByLocalizedDescription : AbstractIndexCreationTask { - public Attributes_ByName() + public Products_ByLocalizedDescription() { - Map = products => from a in products + Map = products => from product in products select new { - // Define the dynamic-index-fields by calling CreateField - // A dynamic-index-field will be generated for each item in the Attributes list - - // For each item, the field name will be the value of field 'PropName' - // The field terms will be derived from field 'PropValue' - _ = a.Attributes.Select(item => CreateField(item.PropName, item.PropValue)), - - // A regular index field can be defined as well: - Name = a.Name + _ = product.Descriptions.Select(x => + CreateField( + "Description_" + x.Key, + x.Value, + new CreateFieldOptions + { + Indexing = FieldIndexing.Search, + Storage = FieldStorage.No + })) }; + + StoreAllFields(FieldStorage.No); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + Analyze(Constants.Documents.Indexing.Fields.AllFields, "StopAnalyzer"); } } -`} - +``` - - -{`public class Attributes_ByName_JS : AbstractJavaScriptIndexCreationTask + +```csharp +public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask { - public Attributes_ByName_JS() + public Products_ByLocalizedDescription_JS() { Maps = new HashSet { - @"map('Products', function (p) { + @"map('Products', function (product) { return { - _: p.Attributes.map(item => createField(item.PropName, item.PropValue, - { indexing: 'Search', storage: true, termVector: null })), - Name: p.Name + _: Object.keys(product.Descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" }; + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions + { + Storage = FieldStorage.No, + Analyzer = "StopAnalyzer" + }; } } -`} - - - - -* **The query**: - * To get all documents matching a specific attribute property use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'Width' is a dynamic-index-field that was indexed from the Attributes list - .WhereEquals("Width", 10) - .ToList(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the Attributes list -from index 'Attributes/ByName' where Width = 10 -`} - +``` + + + + +### Which analyzer is used for the query term? + +The following applies when a `search()` query targets a **dynamic field**: + +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `AllFields` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `AllFields` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `Analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `Analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | + -## CreateField syntax + + + #### Syntax for LINQ-index: - - -{`object CreateField(string name, object value); + +```csharp +object CreateField(string name, object value); object CreateField(string name, object value, bool stored, bool analyzed); object CreateField(string name, object value, CreateFieldOptions options); -`} - +``` #### Syntax for JavaScript-index: - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` | Parameters | Type | Description | |----------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| | **fieldName** | `string` | Name of the dynamic-index-field | -| **fieldValue** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorate.Yes` | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorage.Yes` | | **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | | **options** | `CreateFieldOptions` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| | **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -540,11 +930,15 @@ object CreateField(string name, object value, CreateFieldOptions options); -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. + + + +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) + + \ No newline at end of file diff --git a/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-java.mdx b/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-java.mdx index 17f3cbe360..d54ef6bea8 100644 --- a/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-java.mdx +++ b/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-java.mdx @@ -2,479 +2,918 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -* For such dynamic data, you can define indexes with **dynamic-index-fields**. + -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. + -* Any value type can be indexed, string, number, date, etc. + + +### Example - index every field under an object -* An index definition can contain both dynamic-index-fields and regular-index-fields. + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. + -* In this page: +* **The document**: + + + ```java + public class Product { + private String id; - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) + // The KEYS under the attributes object will be dynamically indexed + // Fields added to this object after index creation time will also get indexed + private Map attributes; - + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "attributes": { + "color": "Red", + "size": 42 + } + } + ``` + -## Indexing documents fields KEYS +* **The index**: + + The following index will index any field under the `attributes` object from the document, + a dynamic-index-field will be created for each such field. + New fields added to the object after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `color` & `size` will become the actual dynamic-index-fields. + + + + ```java + public class Products_ByAttributeKey extends AbstractIndexCreationTask { + public Products_ByAttributeKey() { + + // Call 'CreateField' to generate dynamic-index-fields + // from the attributes object keys. + + // Using '_' is just a convention. + // Any other string can be used instead of '_'. + + // The actual field name will be 'item.Key' + // The actual field terms will be derived from 'item.Value' + map = "docs.Products.Select(p => new { " + + " _ = p.attributes.Select(item => " + + " this.CreateField(item.Key, item.Value)) " + + "})"; + } + } + ``` + + + ```java + public class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByAttributeKey_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + // The field name will be the key + // The field terms will be derived from the corresponding value + " _: Object.keys(p.attributes) " + + " .map(key => createField(key, p.attributes[key])) " + + " }; " + + "})" + )); + } + } + ``` + + -## Example - index any field under object +* **The query**: + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _size_ use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Products_ByAttributeKey.class) + // 'size' is a dynamic-index-field that was indexed from the attributes object + .whereEquals("size", 42) + .toList(); + ``` + + + ```sql + // 'size' is a dynamic-index-field that was indexed from the attributes object + from index 'Products/ByAttributeKey' where size = 42 + ``` + + + + + Dynamic index fields are queried with _DocumentQuery_ or _RQL_ because the field names (e.g. `size`, `color`) + are generated at indexing time and are not properties on the _Product_ class. + + Use the string-based `whereEquals("size", 42)` or RQL `where size = 42` instead. + - + -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + + +### Example - index every field + +The following example allows you to: + * Define an index on a collection **without** needing any common structure between the indexed documents. + * After the index is deployed, any new field added to any document in the collection will be indexed as well. + + + +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`public class Product \{ - private String id; - - // The KEYS under the attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - private Dictionary attributes; - - // get + set implementation ... -\} -`} - - - - -{`// Sample document content -\{ - "attributes": \{ - "color": "Red", - "size": 42 - \} -\} -`} - - + + + ```java + public class Product { + private String id; + + // All KEYS in the document will be dynamically indexed + // Fields added to the document after index creation time will also get indexed + private String firstName; + private String lastName; + private String title; + // ... + + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "firstName": "John", + "lastName": "Doe", + "title": "Engineer" + // ... + } + ``` + * **The index**: - The following index will index any field under the `attributes` object from the document, - a dynamic-index-field will be created for each such field. - New fields added to the object after index creation time will be dynamically indexed as well. + + The following index will index any field from the document, + a dynamic-index-field will be created for each field. + New fields added to the document after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + + The example uses an `AbstractJavaScriptIndexCreationTask` because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a map or collection. + + + + ```java + public class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByAnyField_JS() { + + // This will index EVERY FIELD under the top level of the document + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + " _: Object.keys(p).map(key => createField(key, p[key])) " + + " }; " + + "})" + )); + } + } + ``` + + - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - e.g. Keys `color` & `size` will become the actual dynamic-index-fields. +* **The query**: + + To get all documents with a specific _lastName_ use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Products_ByAnyField_JS.class) + // 'lastName' is a dynamic-index-field that was indexed from the document + .whereEquals("lastName", "Doe") + .toList(); + ``` + + + ```sql + // 'lastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where lastName = "Doe" + ``` + + + + + + - - - -{`public class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { - public Products_ByAttributeKey_JS() { - // Call 'createField' to generate dynamic-index-fields from the attributes object keys - // Using '_' is just a convention. Any other string can be used instead of '_' + - // The actual field name will be the key - // The actual field terms will be derived from p.attributes[key] - setMaps(Sets.newHashSet( - "map('Products', function (p) { " + - " return { " + - " _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], " + - " { indexing: 'Search', storage: false, termVector: null })) " + - " }; " + - "}) " - )); - } -} -`} - - - + + +### Example - basic -* **The query**: - * You can now query the generated dynamic-index fields. - Property `_` is Not queryable, it is only used in the index definition syntax. - * To get all documents with some 'size' use: - - - -{`List matchingDocuments = session - .query(Product.class, Products_ByAttributeKey_JS.class) - .whereEquals("size", 42) - .toList(); -`} - - - - -{`// 'size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey/JS' where size = 42 -`} - - - + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + -## Example - index any field +* **The document**: + + + ```java + public class Product { + private String id; - + // The VALUE of productType will be dynamically indexed + private String productType; + private int pricePerUnit; -* Define an index on a collection **without** needing any common structure between the indexed documents. -* After index is deployed, any new field added to the document will be indexed as well. + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "productType": "Electronics", + "pricePerUnit": 23 + } + ``` + - +* **The index**: + + The following index will index the **value** of document field 'productType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + + ```java + public class Products_ByProductType extends AbstractIndexCreationTask { + public Products_ByProductType() { + + // Call 'CreateField' to generate the dynamic-index-field. + // The field name will be the value of document field 'productType'. + // The field terms will be derived from document field 'pricePerUnit'. + map = "docs.Products.Select(p => new { " + + " _ = this.CreateField(p.productType, p.pricePerUnit) " + + "})"; + } + } + ``` + + + ```java + public class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByProductType_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + // The field name will be the value of document field 'productType' + // The field terms will be derived from document field 'pricePerUnit' + " _: createField(p.productType, p.pricePerUnit) " + + " }; " + + "})" + )); + } + } + ``` + + - -Consider if that is a true necessity, as indexing every single field can end up costing time and disk space. +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Products_ByProductType.class) + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + .whereEquals("Electronics", 23) + .toList(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + + +### Example - list + + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. * **The document**: - - -{`public class Product \{ - private String id; - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - private String firstName; - private String lastName; - private String title; - // ... - - // get + set implementation ... -\} -`} - - - - - -{`// Sample document content -\{ - "firstName": "John", - "lastName": "Doe", - "title": "Engineer", - // ... -\} -`} - - + + + ```java + public class Product { + private String id; + private String name; + + // For each element in this list, + // the VALUE of property 'propName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + private List attributes; + + // getters and setters ... + } -* **The index**: - The following index will index any field from the document, - a dynamic-index-field will be created for each field. - New fields added to the document after index creation time will be dynamically indexed as well. + public class Attribute { + private String propName; + private String propValue; - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "name": "SomeName", + "attributes": [ + { + "propName": "Color", + "propValue": "Blue" + }, + { + "propName": "Width", + "propValue": "10" + }, + { + "propName": "Length", + "propValue": "20" + } + // ... + ] + } + ``` + - - - -{`public class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { - public Products_ByAnyField_JS() { +* **The index**: - // This will index EVERY FIELD under the top level of the document - setMaps(Sets.newHashSet( - "map('Products', function (p) { " + - " return { " + - " _: Object.keys(p).map(key => createField(key, p[key], " + - " { indexing: 'Search', storage: true, termVector: null })) " + - " }; " + - "}) " - )); + The following index will create a dynamic-index-field per item in the document's `attributes` list. + New items added to the attributes list after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the item's propName **value**. + E.g., 'propName' value `Width` will be a dynamic-index-field. + + + + ```java + public class Attributes_ByName extends AbstractIndexCreationTask { + public Attributes_ByName() { + + // Define the dynamic-index-fields by calling 'CreateField'. + // A dynamic-index-field will be generated for each item in 'attributes'. + + // For each item, the field name will be the value of field 'propName' + // The field terms will be derived from field 'propValue'. + // A regular index field (Name) is defined as well. + map = "docs.Products.Select(p => new { " + + " _ = p.attributes.Select(item => " + + " this.CreateField(item.propName, item.propValue)), " + + " Name = p.name " + + "})"; + } } -} -`} - - - + ``` + + + ```java + public class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask { + public Attributes_ByName_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + // For each item, + // the field name will be the value of field 'propName'. + // The field terms will be derived from field 'propValue'. + " _: p.attributes.map(item => " + + " createField(item.propName, item.propValue)), " + + // A regular index field can be defined as well: + " Name: p.name " + + " }; " + + "})" + )); + } + } + ``` + + * **The query**: - * To get all documents with some 'lastName' use: - - - -{`List matchingDocuments = session - .query(Product.class, Products_ByAnyField_JS.class) - .whereEquals("lastName", "Doe") - .toList(); -`} - - - - -{`// 'lastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where lastName = "Doe" -`} - - - + + To get all documents matching a specific attribute property use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Attributes_ByName.class) + // 'Width' is a dynamic-index-field that was indexed from the attributes list + .whereEquals("Width", 10) + .toList(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `indexing: 'Search'` in the `createField` options object (`AbstractJavaScriptIndexCreationTask`) +or `FieldIndexing.Search` in the `CreateFieldOptions` (`AbstractIndexCreationTask`). +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. -## Indexing documents fields VALUES + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). -## Example - basic +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. - + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. -* For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---index-a-list-of-properties) below. + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: -* **The document**: - - -{`public class Product \{ - private String id; - - // The VALUE of productType will be dynamically indexed - private String productType; - private int pricePerUnit; - - // get + set implementation ... -\} -`} - - + + ```java + public class Product { + private String id; - - -{`// Sample document content -\{ - "productType": "Electronics", - "pricePerUnit": 23 -\} -`} - - + private Map descriptions; -* **The index**: - The following index will index the **value** of document field 'productType'. + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `descriptions` map. + For example, the key `English` creates the dynamic field `Description_English`. + + Each generated dynamic field is indexed for **full-text search**. + + + + ```java + public class Products_ByLocalizedDescription extends AbstractIndexCreationTask { + + private static final String USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription() { + map = "docs.Products.Select(product => new { " + + " _ = product.descriptions.Select(x => " + + " this.CreateField( " + + " \"Description_\" + x.Key, " + + " x.Value, " + + " new CreateFieldOptions { " + + // Index each generated dynamic field for FTS + " Indexing = FieldIndexing.Search, " + + " Storage = FieldStorage.No " + + " })) " + + "})"; + + storeAllFields(FieldStorage.NO); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + getConfiguration().put(USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS, "true"); + } + } + ``` + + + ```java + public class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + + private static final String USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (product) { " + + " return { " + + " _: Object.keys(product.descriptions) " + + " .map(function (key) { " + + " return createField( " + + " 'Description_' + key, " + + " product.descriptions[key], " + + " { " + + // Index each generated dynamic field for FTS + " indexing: 'Search', " + + " storage: false " + + " }); " + + " }) " + + " }; " + + "})" + )); + + Map fields = new HashMap<>(); + + IndexFieldOptions allFieldsOptions = new IndexFieldOptions(); + allFieldsOptions.setStorage(FieldStorage.NO); + + fields.put(Constants.Documents.Indexing.Fields.ALL_FIELDS, allFieldsOptions); + setFields(fields); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + getConfiguration().put(USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS, "true"); + } + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```java + List results = session.advanced() + .documentQuery(Product.class, Products_ByLocalizedDescription.class) + .search("Description_English", "north wind") + .toList(); + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + - This value will be the dynamic-index-field name on which you can query. - e.g. Field value `Electronics` will be the dynamic-index-field. +--- - - - -{`public class Products_ByProductType extends AbstractIndexCreationTask { - public Products_ByProductType() { + + +### Configuring analyzers for dynamic fields + +The `CreateFieldOptions` (and the JavaScript `createField` options) do **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: - // The field name will be the value of document field 'productType' - // The field terms will be derived from document field 'pricePerUnit' - map = "docs.Products.Select(p => new { " + - " _ = this.CreateField(p.productType, p.pricePerUnit) " + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field + + + +```java +public class Products_ByLocalizedDescription extends AbstractIndexCreationTask { + public Products_ByLocalizedDescription() { + map = "docs.Products.Select(product => new { " + + " _ = product.descriptions.Select(x => " + + " this.CreateField( " + + " \"Description_\" + x.Key, " + + " x.Value, " + + " new CreateFieldOptions { " + + " Indexing = FieldIndexing.Search, " + + " Storage = FieldStorage.No " + + " })) " + "})"; + + storeAllFields(FieldStorage.NO); + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`List matchingDocuments = session - .query(Product.class, Products_ByProductType.class) - .whereEquals("Electronics", 23) - .toList(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'productType' -from index 'Products/ByProductType' where Electronics = 23 -`} - - - - -## Example - list + +```java +public class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByLocalizedDescription_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (product) { " + + " return { " + + " _: Object.keys(product.descriptions) " + + " .map(function (key) { " + + " return createField( " + + " 'Description_' + key, " + + " product.descriptions[key], " + + " { " + + " indexing: 'Search', " + + " storage: false " + + " }); " + + " }) " + + " }; " + + "})" + )); - + Map fields = new HashMap<>(); -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. + IndexFieldOptions allFieldsOptions = new IndexFieldOptions(); + allFieldsOptions.setStorage(FieldStorage.NO); + fields.put(Constants.Documents.Indexing.Fields.ALL_FIELDS, allFieldsOptions); - + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + IndexFieldOptions englishOptions = new IndexFieldOptions(); + englishOptions.setAnalyzer("StopAnalyzer"); + fields.put("Description_English", englishOptions); -* **The document**: - - -{`public class Product \{ - private String id; - private String name; - - // For each element in this list, the VALUE of property 'propName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - private List attributes; - - // get + set implementation ... -\} - -public class Attribute \{ - private String propName; - private String propValue; - - // get + set implementation ... -\} -`} - + setFields(fields); + } +} +``` + - - -{`// Sample document content -\{ -name": "SomeName", -attributes": [ - \{ - "propName": "Color", - "propValue": "Blue" - \}, - \{ - "propName": "Width", - "propValue": "10" - \}, - \{ - "propName": "Length", - "propValue": "20" - \}, - ... - -\} -`} - - +#### Configure a fallback analyzer for all fields + + + +```java +public class Products_ByLocalizedDescription extends AbstractIndexCreationTask { + public Products_ByLocalizedDescription() { + map = "docs.Products.Select(product => new { " + + " _ = product.descriptions.Select(x => " + + " this.CreateField( " + + " \"Description_\" + x.Key, " + + " x.Value, " + + " new CreateFieldOptions { " + + " Indexing = FieldIndexing.Search, " + + " Storage = FieldStorage.No " + + " })) " + + "})"; -* **The index**: - The following index will create a dynamic-index-field per item in the document's `attributes` list. - New items added to the attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's propName **value**. - E.g., 'propName' value `Width` will be a dynamic-index-field. - - - - -{`public class Attributes_ByName extends AbstractIndexCreationTask { - public Attributes_ByName() { - - // For each attribute item, the field name will be the value of field 'propName' - // The field terms will be derived from field 'propValue' - // A regular-index-field (Name) is defined as well - map = - "docs.Products.Select(p => new { " + - " _ = p.attributes.Select(item => this.CreateField(item.propName, item.propValue)), " + - " Name = p.name " + - "})"; + storeAllFields(FieldStorage.NO); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + analyze(Constants.Documents.Indexing.Fields.ALL_FIELDS, "StopAnalyzer"); } } -`} - +``` - + +```java +public class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByLocalizedDescription_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (product) { " + + " return { " + + " _: Object.keys(product.descriptions) " + + " .map(function (key) { " + + " return createField( " + + " 'Description_' + key, " + + " product.descriptions[key], " + + " { " + + " indexing: 'Search', " + + " storage: false " + + " }); " + + " }) " + + " }; " + + "})" + )); -* **The query**: - * To get all documents matching a specific attribute property use: - - - -{`List matchingDocuments = session - .query(Product.class, Attributes_ByName.class) - .whereEquals("Width", 10) - .toList(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the attributes list -from index 'Attributes/ByName' where Width = 10 -`} - + Map fields = new HashMap<>(); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + IndexFieldOptions allFieldsOptions = new IndexFieldOptions(); + allFieldsOptions.setStorage(FieldStorage.NO); + allFieldsOptions.setAnalyzer("StopAnalyzer"); + fields.put(Constants.Documents.Indexing.Fields.ALL_FIELDS, allFieldsOptions); + + setFields(fields); + } +} +``` + + + + +### Which analyzer is used for the query term? +The following applies when a `search()` query targets a **dynamic field**: -## CreateField syntax +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | -#### Syntax for LINQ-index: + - - -{`object CreateField(string name, object value); + + + + +#### Syntax for `AbstractIndexCreationTask` + + +```csharp +object CreateField(string name, object value); object CreateField(string name, object value, bool stored, bool analyzed); object CreateField(string name, object value, CreateFieldOptions options); -`} - +``` -#### Syntax for JavaScript-index: +#### Syntax for `AbstractJavaScriptIndexCreationTask` - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` -| Parameters | Type | Description | -|----------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **fieldName** | `string` | Name of the dynamic-index-field | -| **fieldValue** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorate.Yes` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | +| Parameters | Type | Description | +|----------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorage.Yes` | +| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | | **options** | `CreateFieldOptions` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| | **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +* The `CreateField` syntax (and the `CreateFieldOptions` shape) above describes what is available inside the server-side `map` string of an `AbstractIndexCreationTask`, + since the Java client sends the `map` source to the server for compilation. + The JavaScript `createField` function is the equivalent for `AbstractJavaScriptIndexCreationTask` map sources. + * All above examples have used the character `_` in the dynamic-index-field definition. However, using `_` is just a convention. Any other string can be used instead. * This property is Not queryable, it is only used in the index definition syntax. - The actual dynamic-index-fields that are generated are defined by the `CreateField` method. + The actual dynamic-index-fields that are generated are defined by the `CreateField` method (or the `createField` JS function). + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. - +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-nodejs.mdx b/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-nodejs.mdx index b5df2df4a7..c14184c2b5 100644 --- a/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-nodejs.mdx +++ b/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-nodejs.mdx @@ -2,533 +2,710 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. - -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. - -* In this page: - - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`createField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -## Indexing documents fields KEYS + - -#### Example - index any field under object -The following allows you to: - -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. -**The document**: - - -{`class Product \{ - constructor(id, attributes) \{ - this.id = id; - - // The KEYS under the attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - this.attributes = attributes; - \} -\} -`} - - + + +### Example - index every field under an object - - -{`// Sample document content -\{ - "attributes": \{ - "color": "Red", - "size": 42 - \} -\} -`} - - + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. + -**The index**: +* **The document**: + + + ```js + class Product { + constructor(id, attributes) { + this.id = id; + + // The KEYS under the attributes object will be dynamically indexed + // Fields added to this object after index creation time will also get indexed + this.attributes = attributes; + } + } + ``` + + + + ```json + // Sample document content + { + "attributes": { + "color": "Red", + "size": 42 + } + } + ``` + -* The following index will index any field under the `attributes` object from the document, +* **The index**: + + The following index will index any field under the `attributes` object from the document, a dynamic-index-field will be created for each such field. - New fields added to the object after index creation time will be dynamically indexed as well. - -* The actual dynamic-index-field name on which you can query will be the attribute field **key**. - e.g. Keys `color` & `size` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { - constructor() { - super(); - - const { createField } = this.mapUtils(); - - this.map("Products", p => { - return { - // Call 'createField' to generate dynamic-index-fields from the attributes object keys - // Using '_' is just a convention. Any other string can be used instead of '_' - - // The actual field name will be the key - // The actual field terms will be derived from p.attributes[key] - _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], { - indexing: "Search", - storage: false, - termVector: null - })) - }; - }); + New fields added to the object after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `color` & `size` will become the actual dynamic-index-fields. + + + ```js + class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + this.map("Products", p => { + return { + // Call 'createField' to generate dynamic-index-fields + // from the attributes object keys. + + // Using '_' is just a convention. + // Any other string can be used instead of '_' + + // The actual field name will be the key + // The actual field terms will be derived from p.attributes[key] + _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key])) + }; + }); + } } -} -`} - - - - -**The query**: - -* You can now query the generated dynamic-index fields. - Property `_` is Not queryable, it is only used in the index definition syntax. - -* To get all documents with some 'size' use: - - - - -{`const matchingDocuments = session.query({indexName: 'Products_ByAttributeKey'}) - // 'size' is a dynamic-index-field that was indexed from the attributes object - .whereEquals('size', 42) - .all(); -`} - - - - -{`// 'size' is a dynamic-index-field that was indexed from the attributes object -from index 'Products/ByAttributeKey' where size = 42 -`} - - - - - + ``` + + +* **The query**: + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _size_ use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Products/ByAttributeKey/JS" }) + // 'size' is a dynamic-index-field that was indexed from the attributes object + .whereEquals("size", 42) + .all(); + ``` + + + ```sql + // 'size' is a dynamic-index-field that was indexed from the attributes object + from index 'Products/ByAttributeKey/JS' where size = 42 + ``` + + + + + + + +### Example - index every field -#### Example - index any field -The following allows you to: - -* Define an index on a collection **without** needing any common structure between the indexed documents. -* After index is deployed, any new field added to the document will be indexed as well. - +The following example allows you to: + * Define an index on a collection **without** needing any common structure between the indexed documents. + * After the index is deployed, any new field added to any document in the collection will be indexed as well. + + - -Consider if that is a true necessity, as indexing every single field can end up costing time and disk space. - +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. -**The document**: - - -{`class Product \{ - constructor(id, firstName, lastName, title) \{ - this.id = id; - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - this.firstName = firstName; - this.lastName = lastName; - this.title = title; - // ... - \} -\} -`} - - - - - -{`// Sample document content -\{ - "firstName": "John", - "lastName": "Doe", - "title": "Engineer", - // ... -\} -`} - - -**The index**: +* **The document**: + + + ```js + class Product { + constructor(id, firstName, lastName, title) { + this.id = id; + + // All KEYS in the document will be dynamically indexed + // Fields added to the document after index creation time will also get indexed + this.firstName = firstName; + this.lastName = lastName; + this.title = title; + // ... + } + } + ``` + + + + ```json + // Sample document content + { + "firstName": "John", + "lastName": "Doe", + "title": "Engineer", + // ... + } + ``` + -* The following index will index any field from the document, +* **The index**: + + The following index will index any field from the document, a dynamic-index-field will be created for each field. - New fields added to the document after index creation time will be dynamically indexed as well. - -* The actual dynamic-index-field name on which you can query will be the field **key**. - e.g. Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + New fields added to the document after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + + + ```js + class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + // This will index EVERY FIELD under the top level of the document + this.map("Products", p => { + return { + _: Object.keys(p).map(key => createField(key, p[key])) + }; + }); + } + } + ``` + + +* **The query**: + + To get all documents with a specific _lastName_ use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Products/ByAnyField/JS" }) + // 'lastName' is a dynamic-index-field that was indexed from the document + .whereEquals("lastName", "Doe") + .all(); + ``` + + + ```sql + // 'lastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where lastName = "Doe" + ``` + + + + + + + + + + + +### Example - basic + + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + - - - -{`class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { - constructor () { - super(); +* **The document**: + + + ```js + class Product { + constructor(id, productType, pricePerUnit) { + this.id = id; + + // The VALUE of productType will be dynamically indexed + this.productType = productType; + this.pricePerUnit = pricePerUnit; + } + } + ``` + + + + ```json + // Sample document content + { + "productType": "Electronics", + "pricePerUnit": 23 + } + ``` + + +* **The index**: + + The following index will index the **value** of document field 'productType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + ```js + class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + this.map("Products", p => { + return { + // The field name will be the value of document field 'productType' + // The field terms will be derived from document field 'pricePerUnit' + _: createField(p.productType, p.pricePerUnit) + }; + }); + } + } + ``` + + +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Products/ByProductType/JS" }) + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + .whereEquals("Electronics", 23) + .all(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + from index 'Products/ByProductType/JS' where Electronics = 23 + ``` + + + + + + + +### Example - list - const { createField } = this.mapUtils(); + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. + - this.map("Products", p => { - return { - // This will index EVERY FIELD under the top level of the document - _: Object.keys(p).map(key => createField(key, p[key], { - indexing: "Search", - storage: true, - termVector: null - })) - }; - }); +* **The document**: + + + ```js + class Product { + constructor(id, name, attributes) { + this.id = id; + this.name = name; + + // For each element in this list, + // the VALUE of property 'propName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + this.attributes = attributes; + } } -} -`} - - - -**The query**: + class Attribute { + constructor(propName, propValue) { + this.propName = propName; + this.propValue = propValue; + } + } + ``` + + + + ```json + // Sample document content + { + "name": "SomeName", + "attributes": [ + { + "propName": "Color", + "propValue": "Blue" + }, + { + "propName": "Width", + "propValue": "10" + }, + { + "propName": "Length", + "propValue": "20" + }, + ... + ] + } + ``` + + +* **The index**: + + The following index will create a dynamic-index-field per item in the document's `attributes` list. + New items added to the attributes list after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the item's propName **value**. + E.g., 'propName' value `Width` will be a dynamic-index-field. + + + ```js + class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + this.map("Products", p => { + return { + // Define the dynamic-index-fields by calling 'createField' + // A dynamic-index-field will be generated for each item in 'attributes' + + // For each item, the field name will be the value of field 'propName' + // The field terms will be derived from field 'propValue' + _: p.attributes.map(item => + createField(item.propName, item.propValue)), + + // A regular index field can be defined as well: + name: p.name + }; + }); + } + } + ``` + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Attributes/ByName/JS" }) + // 'Width' is a dynamic-index-field that was indexed from the attributes list + .whereEquals("Width", 10) + .all(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the attributes list + from index 'Attributes/ByName/JS' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `indexing: "Search"` in the `createField` options. + +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. + + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. + + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. -* To get all documents with some 'lastName' use: + + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```js + class Product { + constructor(id, descriptions) { + this.id = id; + this.descriptions = descriptions; + } + } + ``` + + + + ```json + // Sample document content + { + "descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. + + Each generated dynamic field is indexed for **full-text search**. + + + ```js + class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + const useSearchAnalyzerForDynamicFields = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + this.map("Products", product => { + return { + _: Object.keys(product.descriptions).map(key => + createField( + "Description_" + key, + product.descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: "Search", + storage: false + })) + }; + }); + + this.storeAllFields("No"); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + this.configuration[useSearchAnalyzerForDynamicFields] = "true"; + } + } + ``` + - - - -{`const matchingDocuments = session.query({ indexName: 'Products_ByAnyField_JS' }) - // 'lastName' is a dynamic-index-field that was indexed from the document - .whereEquals('lastName', 'Doe') - .all(); -`} - - - - -{`// 'lastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where lastName = "Doe" -`} - - - +* **Full-text search query**: - + Query the generated dynamic field by its field name. + In this example, the query targets the generated field `Description_English`. + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). -## Indexing documents fields VALUES + + + ```js + const results = await session + .query({ indexName: "Products/ByLocalizedDescription/JS" }) + .search("Description_English", "north wind") + .all(); + ``` + - -#### Example - basic -This example shows: - -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. -* For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---index-a-list-of-properties) below. -**The document**: - - -{`class Product \{ - constructor(id, productType, pricePerUnit) \{ - this.id = id; - - // The VALUE of productType will be dynamically indexed - this.productType = productType; - this.pricePerUnit = pricePerUnit; - \} -\} -`} - - + + ```sql + from index "Products/ByLocalizedDescription/JS" + where search(Description_English, "north wind") + ``` + + - - -{`// Sample document content -\{ - "productType": "Electronics", - "pricePerUnit": 23 -\} -`} - - +--- -**The index**: + + +### Configuring analyzers for dynamic fields -* The following index will index the **value** of document field 'productType'. +The `createField` options object does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. -* This value will be the dynamic-index-field name on which you can query. - e.g. Field value `Electronics` will be the dynamic-index-field. +--- - - - -{`class Products_ByProductType extends AbstractCsharpIndexCreationTask { - constructor () { - super(); +#### Configure analyzer for a specific dynamic field - // The field name will be the value of document field 'productType' - // The field terms will be derived from document field 'pricePerUnit' - this.map = "docs.Products.Select(p => new { " + - " _ = this.CreateField(p.productType, p.pricePerUnit) " + - "})"; - } -} -`} - - - - -{`class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask { - constructor () { + +```js +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { super(); const { createField } = this.mapUtils(); - this.map("Products", p => { + this.map("Products", product => { return { - _: [ - // The field name will be the value of document field 'productType' - // The field terms will be derived from document field 'pricePerUnit' - createField(p.productType, p.pricePerUnit, { - indexing: "Search", - storage: false, - termVector: null - }) - ] + _: Object.keys(product.descriptions).map(key => + createField( + "Description_" + key, + product.descriptions[key], + { + indexing: "Search", + storage: false + })) }; }); - } -} -`} - - - - -**The query**: - -* To get all documents of some product type having a specific price per unit use: - - - - -{`const matchingDocuments = session.query({ indexName: 'Products_ByProductType' }) - // 'Electronics' is the dynamic-index-field that was indexed from document field 'productType' - .whereEquals('Electronics', 23) - .all(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'productType' -from index 'Products/ByProductType' where Electronics = 23 -`} - - - - - - -#### Example - list -The following allows you to: - -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. -**The document**: - - -{`class Product \{ - constructor(id, name, attributes) \{ - this.id = id; - this.name = name; - - // For each element in this list, the VALUE of property 'propName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - this.attributes = attributes; - \} -\} - -class Attribute \{ - constructor(propName, propValue) \{ - this.propName = propName; - this.propValue = propValue; - \} -\} -`} - - - - - -{`// Sample document content -\{ - "name": "SomeName", - "attributes": [ - \{ - "propName": "Color", - "propValue": "Blue" - \}, - \{ - "propName": "Width", - "propValue": "10" - \}, - \{ - "propName": "Length", - "propValue": "20" - \}, - ... - ] -\} -`} - - - -**The index**: - -* The following index will create a dynamic-index-field per item in the document's `attributes` list. - New items added to the attributes list after index creation time will be dynamically indexed as well. - -* The actual dynamic-index-field name on which you can query will be the item's propName **value**. - e.g. 'propName' value `Width` will be a dynamic-index-field. - - - - -{`class Attributes_ByName extends AbstractCsharpIndexCreationTask -{ - constructor () { - super(); + this.storeAllFields("No"); - // For each attribute item, the field name will be the value of field 'propName' - // The field terms will be derived from field 'propValue' - // A regular-index-field (Name) is defined as well - this.map = - "docs.Products.Select(p => new { " + - " _ = p.attributes.Select(item => this.CreateField(item.propName, item.propValue)), " + - " Name = p.name " + - "})"; + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + this.analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -{`class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask { - constructor () { + +#### Configure a fallback analyzer for all fields + + +```js +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { super(); const { createField } = this.mapUtils(); - this.map("Products", p => { + this.map("Products", product => { return { - // For each item, the field name will be the value of field 'propName' - // The field terms will be derived from field 'propValue' - _: p.attributes.map(item => createField(item.propName, item.propValue, { - indexing: "Search", - storage: true, - termVector: null - })), - - // A regular-index-field can be defined as well: - Name: p.name + _: Object.keys(product.descriptions).map(key => + createField( + "Description_" + key, + product.descriptions[key], + { + indexing: "Search", + storage: false + })) }; }); - } -} -`} - - - - -**The query**: -* To get all documents matching a specific attribute property use: + this.storeAllFields("No"); - - - -{`const matchingDocuments = session.query({ indexName: 'Attributes/ByName' }) - // 'Width' is a dynamic-index-field that was indexed from the attributes list - .whereEquals('Width', 10) - .all(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the attributes list -from index 'Attributes/ByName' where Width = 10 -`} - + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + this.analyze(CONSTANTS.Documents.Indexing.Fields.ALL_FIELDS, "StopAnalyzer"); + } +} +``` - - - + + + + +### Which analyzer is used for the query term? -## CreateField syntax +The following applies when a `search()` query targets a **dynamic field**: -#### Syntax for LINQ-index: +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `this.analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `this.analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | - - -{`object CreateField(string name, object value); + -object CreateField(string name, object value, bool stored, bool analyzed); + -object CreateField(string name, object value, CreateFieldOptions options); -`} - - + -#### Syntax for JavaScript-index: +#### Syntax: - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` -| Parameters | Type | Description | -|----------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **fieldName** | `string` | Name of the dynamic-index-field | -| **fieldValue** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorate.Yes` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | -| **options** | `CreateFieldOptions` | Dynamic-index-field options | +| Parameters | Type | Description | +|----------------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **options** | `object` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| -| **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| options object | | | +|-----------------|--------------------|-----------------------------------------------------------------------------------| +| **storage** | `boolean` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | +| **indexing** | `FieldIndexing` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **termVector** | `FieldTermVector` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -536,21 +713,19 @@ object CreateField(string name, object value, CreateFieldOptions options); However, using `_` is just a convention. Any other string can be used instead. * This property is Not queryable, it is only used in the index definition syntax. - The actual dynamic-index-fields that are generated are defined by the `CreateField` method. + The actual dynamic-index-fields that are generated are defined by the `createField` method. + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. - +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-php.mdx b/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-php.mdx index 09c58cc9fc..6992310715 100644 --- a/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-php.mdx +++ b/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-php.mdx @@ -2,537 +2,969 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. + -* In this page: + - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) + + +### Example - index every field under an object + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. -## Indexing documents fields KEYS - -## Example - index any field under object +* **The document**: + + + ```php + use Ds\Map as DSMap; - + class Product + { + public ?string $id = null; -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + // The KEYS under the attributes object will be dynamically indexed. + // Fields added to this object after index creation time will also get indexed. + public ?DSMap $attributes = null; - - -* **The document**: - - -{`use Ds\\Map as DSMap; - -class Product -\{ - private ?string $id = null; - - // The KEYS under the Attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - public ?DSMap $attributes = null; -\} -`} - - - - -{`// Sample document content -\{ - "Attributes": \{ - "Color": "Red", - "Size": 42 - \} -\} -`} - - + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "attributes": { + "Color": "Red", + "Size": 42 + } + } + ``` + * **The index**: - The below index will index any field under the `Attributes` object from the document, + + The following index will index any field under the `attributes` object from the document, a dynamic-index-field will be created for each such field. New fields added to the object after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. - - - -{`class Products_ByAttributeKey extends AbstractIndexCreationTask -{ - public function __construct() + + + ```php + class Products_ByAttributeKey extends AbstractIndexCreationTask { - parent::__construct(); - - $this->map = "from p in docs.Products select new {" . - "_ = p.attributes.Select(item => CreateField(item.Key, item.Value))" . - "}"; + public function __construct() + { + parent::__construct(); + + // Call 'CreateField' to generate dynamic-index-fields + // from the attributes object keys. + // + // Using '_' is just a convention. + // Any other string can be used instead of '_'. + // + // The actual field name will be 'item.Key'. + // The actual field terms will be derived from 'item.Value'. + $this->map = + "from p in docs.Products " . + "select new { " . + " _ = p.attributes.Select(item => CreateField(item.Key, item.Value)) " . + "}"; + } } -} -`} - - - - -{`class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask -{ - public function __construct() + ``` + + + ```php + class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { - parent::__construct(); - - $this->setMaps([ - "map('Products', function (p) { " . - " return { " . - " _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], " . - " { indexing: 'Search', storage: false, termVector: null })) " . - " }; " . - "}) " - ]); + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (p) { + return { + // The field name will be the key + // The field terms will be derived from the corresponding value + _: Object.keys(p.attributes) + .map(key => createField(key, p.attributes[key], null)) + }; + })" + ]); + } } -} -`} - - - - + ``` + + * **The query**: - * You can now query the generated dynamic-index fields. - * To get all documents with some 'size' use: - - - -{`$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Products_ByAttributeKey::class) - // 'Size' is a dynamic-index-field that was indexed from the Attributes object - ->whereEquals("Size", 42) - ->toList(); -`} - - - - -{`// 'Size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey' where Size = 42 -`} - - - - -## Example - index any field - - + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _Size_ use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Products_ByAttributeKey::class) + // 'Size' is a dynamic-index-field that was indexed from the attributes object + ->whereEquals("Size", 42) + ->toList(); + ``` + + + ```sql + // 'Size' is a dynamic-index-field that was indexed from the attributes object + from index 'Products/ByAttributeKey' where Size = 42 + ``` + + + + + Dynamic index fields are queried with _documentQuery_ or _RQL_ because the field names (e.g. `Size`, `Color`) + are generated at indexing time and are not properties on the _Product_ class. + + Use the string-based `whereEquals("Size", 42)` or RQL `where Size = 42` instead. + + + + + + +### Example - index every field + +The following example allows you to: * Define an index on a collection **without** needing any common structure between the indexed documents. - * After index is deployed, any new field added to the document will be indexed as well. - + * After the index is deployed, any new field added to any document in the collection will be indexed as well. - + -Consider whether this is really necessary, as indexing every single field can end up costing time and disk space. +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`class Product -\{ - private ?string $id = null; - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - public ?string $firstName = null; - public ?string $lastName = null; - public ?string $title = null; - - // ... getters and setters -\} -`} - - + + + ```php + class Product + { + public ?string $id = null; - - -{`// Sample document content - \{ - "FirstName": "John", - "LastName": "Doe", - "Title": "Engineer", + // All KEYS in the document will be dynamically indexed. + // Fields added to the document after index creation time will also get indexed. + public ?string $firstName = null; + public ?string $lastName = null; + public ?string $title = null; // ... -\} -`} - - + + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "firstName": "John", + "lastName": "Doe", + "title": "Engineer" + // ... + } + ``` + * **The index**: - The below index will index any field from the document, + + The following index will index any field from the document, a dynamic-index-field will be created for each field. New fields added to the document after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask -{ - public function __construct() + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + + The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection. + + + + ```php + class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { - parent::__construct(); - - // This will index EVERY FIELD under the top level of the document - $this->setMaps([ - "map('Products', function (p) { - return { - _: Object.keys(p).map(key => createField(key, p[key], - { indexing: 'Search', storage: true, termVector: null })) - } - })" - ]); + public function __construct() + { + parent::__construct(); + + // This will index EVERY FIELD under the top level of the document + $this->setMaps([ + "map('Products', function (p) { + return { + _: Object.keys(p).map(key => createField(key, p[key], null)) + }; + })" + ]); + } } -} -`} - - - + ``` + + * **The query**: - * To get all documents with some 'LastName' use: - - - -{`$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Products_ByAnyField_JS::class) - // 'LastName' is a dynamic-index-field that was indexed from the document - ->whereEquals("LastName", "Doe") - ->toList(); -`} - - - - -{`// 'LastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where LastName = "Doe" -`} - - - + + To get all documents with a specific _lastName_ use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Products_ByAnyField_JS::class) + // 'lastName' is a dynamic-index-field that was indexed from the document + ->whereEquals("lastName", "Doe") + ->toList(); + ``` + + + ```sql + // 'lastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where lastName = "Doe" + ``` + + + + + + + + + + + +### Example - basic + + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + +* **The document**: + + + ```php + class Product + { + public ?string $id = null; + // The VALUE of productType will be dynamically indexed + public ?string $productType = null; + public ?int $pricePerUnit = null; -## Indexing documents fields VALUES + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "productType": "Electronics", + "pricePerUnit": 23 + } + ``` + -## Example - basic +* **The index**: + + The following index will index the **value** of document field 'productType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. - + + + ```php + class Products_ByProductType extends AbstractIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + // Call 'CreateField' to generate the dynamic-index-fields. + // The field name will be the value of document field 'productType'. + // The field terms will be derived from document field 'pricePerUnit'. + $this->map = + "from p in docs.Products " . + "select new { " . + " _ = CreateField(p.productType, p.pricePerUnit) " . + "}"; + } + } + ``` + + + ```php + class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (p) { + return { + // The field name will be the value of document field 'productType' + // The field terms will be derived from document field 'pricePerUnit' + _: createField(p.productType, p.pricePerUnit, null) + }; + })" + ]); + } + } + ``` + + -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Products_ByProductType::class) + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + ->whereEquals("Electronics", 23) + ->toList(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + + +### Example - list + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. * **The document**: - - -{`class Product -\{ - public ?string $id = null; - - // The VALUE of ProductType will be dynamically indexed - public ?string $productType = null; - public ?int $pricePerUnit = null; - - // ... getters and setters -\} -`} - - + + + ```php + class Product + { + public ?string $id = null; + public ?string $name = null; - - -{`// Sample document content -\{ - "ProductType": "Electronics", - "PricePerUnit": 23 -\} -`} - - + // For each element in this list, + // the VALUE of property 'propName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + public ?AttributeList $attributes = null; + + // ... getters and setters + } + + class Attribute + { + public ?string $propName = null; + public ?string $propValue = null; + + // ... getters and setters + } + + class AttributeList extends TypedList + { + protected function __construct() + { + parent::__construct(Attribute::class); + } + } + ``` + + + + ```json + // Sample document content + { + "name": "SomeName", + "attributes": [ + { + "propName": "Color", + "propValue": "Blue" + }, + { + "propName": "Width", + "propValue": "10" + }, + { + "propName": "Length", + "propValue": "20" + } + // ... + ] + } + ``` + * **The index**: - The below index will index the **value** of document field 'ProductType'. + + The following index will create a dynamic-index-field per item in the document's `attributes` list. + New items added to the attributes list after index creation time will be dynamically indexed as well. - This value will be the dynamic-index-field name on which you can query. - E.g., Field value `Electronics` will be the dynamic-index-field. + The actual dynamic-index-field name on which you can query will be the item's propName **value**. + E.g., 'propName' value `Width` will be a dynamic-index-field. + + + + ```php + class Attributes_ByName extends AbstractIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + // Define the dynamic-index-fields by calling 'CreateField'. + // A dynamic-index-field will be generated for each item in 'attributes'. + // + // For each item, the field name will be the value of field 'propName'. + // The field terms will be derived from field 'propValue'. + $this->map = + "from p in docs.Products " . + "select new { " . + " _ = p.attributes.Select(item => CreateField(item.propName, item.propValue)), " . + " Name = p.name " . + "}"; + } + } + ``` + + + ```php + class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (p) { + return { + // For each item, + // the field name will be the value of field 'propName' + // The field terms will be derived from field 'propValue' + _: p.attributes.map(item => createField( + item.propName, item.propValue, null)), + + // A regular index field can be defined as well: + Name: p.name + }; + })" + ]); + } + } + ``` + + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Attributes_ByName::class) + // 'Width' is a dynamic-index-field that was indexed from the attributes list + ->whereEquals("Width", 10) + ->toList(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `FieldIndexing::search()` in `CreateFieldOptions`. + +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. + + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. + + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + + + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```php + use Ds\Map as DSMap; + + class Product + { + public ?string $id = null; + public ?DSMap $descriptions = null; + + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. - + Each generated dynamic field is indexed for **full-text search**. + + + + ```php + class Products_ByLocalizedDescription extends AbstractIndexCreationTask + { + private const USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public function __construct() + { + parent::__construct(); + + $this->map = + "from product in docs.Products " . + "select new { " . + " _ = product.descriptions.Select(x => " . + " CreateField(" . + " \"Description_\" + x.Key, " . + " x.Value, " . + " new CreateFieldOptions { " . + // Index each generated dynamic field for FTS + " Indexing = FieldIndexing.Search, " . + " Storage = FieldStorage.No " . + " })) " . + "}"; + + $this->storeAllFields(FieldStorage::no()); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + $this->getConfiguration()[self::USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS] = "true"; + } + } + ``` + + + ```php + class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask + { + private const USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (product) { + return { + _: Object.keys(product.descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: 'Search', + storage: 'No' + }); + }) + }; + })" + ]); + + // Apply a storage default to every generated field + $fields = new IndexFieldOptionsArray(); + $allFieldsOptions = new IndexFieldOptions(); + $allFieldsOptions->setStorage(FieldStorage::no()); + $fields->offsetSet(DocumentsIndexingFields::ALL_FIELDS, $allFieldsOptions); + $this->setFields($fields); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + $this->getConfiguration()[self::USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS] = "true"; + } + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```php + /** @var array $results */ + $results = $session->advanced() + ->documentQuery(Product::class, Products_ByLocalizedDescription::class) + ->search("Description_English", "north wind") + ->toList(); + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + + +--- + + + +### Configuring analyzers for dynamic fields + +`CreateFieldOptions` does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field + + - -{`class Products_ByProductType extends AbstractIndexCreationTask +```php +class Products_ByLocalizedDescription extends AbstractIndexCreationTask { public function __construct() { parent::__construct(); - // Call 'CreateField' to generate the dynamic-index-fields - // The field name will be the value of document field 'ProductType' - // The field terms will be derived from document field 'PricePerUnit' - $this->map = "docs.Products.Select(p => new { " . - " _ = this.CreateField(p.productType, p.pricePerUnit) " . - "})"; + $this->map = + "from product in docs.Products " . + "select new { " . + " _ = product.descriptions.Select(x => " . + " CreateField(" . + " \"Description_\" + x.Key, " . + " x.Value, " . + " new CreateFieldOptions { " . + " Indexing = FieldIndexing.Search, " . + " Storage = FieldStorage.No " . + " })) " . + "}"; + + $this->storeAllFields(FieldStorage::no()); + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + $this->analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -{`class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask + +```php +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { public function __construct() { parent::__construct(); $this->setMaps([ - "map('Products', function (p) { + "map('Products', function (product) { return { - _: createField(p.ProductType, p.PricePerUnit, - { indexing: 'Search', storage: true, termVector: null }) + _: Object.keys(product.descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" ]); - } -} -`} - - - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Products_ByProductType::class) -// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -->whereEquals("Electronics", 23) -->toList(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -from index 'Products/ByProductType' where Electronics = 23 -`} - - - - -## Example - list - - -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. + // Apply a storage default to every generated field + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + $fields = new IndexFieldOptionsArray(); - + $allFieldsOptions = new IndexFieldOptions(); + $allFieldsOptions->setStorage(FieldStorage::no()); + $fields->offsetSet(DocumentsIndexingFields::ALL_FIELDS, $allFieldsOptions); -* **The document**: - - -{`class Product -\{ - public ?string $id = null; - public ?string $name = null; - - // For each element in this list, the VALUE of property 'PropName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - public ?AttributeList $attributes = null; - - // ... getters and setters -\} - -class Attribute -\{ - public ?string $propName = null; - public ?string $propValue = null; - - // ... getters and setters -\} - -class AttributeList extends TypedList -\{ - protected function __construct() - \{ - parent::__construct(Attribute::class); - \} -\} -`} - - + $englishFieldOptions = new IndexFieldOptions(); + $englishFieldOptions->setAnalyzer("StopAnalyzer"); + $fields->offsetSet("Description_English", $englishFieldOptions); - - -{`// Sample document content -\{ -Name": "SomeName", -Attributes": [ - \{ - "PropName": "Color", - "PropValue": "Blue" - \}, - \{ - "PropName": "Width", - "PropValue": "10" - \}, - \{ - "PropName": "Length", - "PropValue": "20" - \}, - ... - -\} -`} - + $this->setFields($fields); + } +} +``` + -* **The index**: - The below index will create a dynamic-index-field per item in the document's `Attributes` list. - New items added to the Attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's PropName **value**. - E.g., 'PropName' value `width` will be a dynamic-index-field. +#### Configure a fallback analyzer for all fields - + - -{`class Attributes_ByName extends AbstractIndexCreationTask +```php +class Products_ByLocalizedDescription extends AbstractIndexCreationTask { public function __construct() { parent::__construct(); - // Define the dynamic-index-fields by calling CreateField - // A dynamic-index-field will be generated for each item in the Attributes list + $this->map = + "from product in docs.Products " . + "select new { " . + " _ = product.descriptions.Select(x => " . + " CreateField(" . + " \"Description_\" + x.Key, " . + " x.Value, " . + " new CreateFieldOptions { " . + " Indexing = FieldIndexing.Search, " . + " Storage = FieldStorage.No " . + " })) " . + "}"; - // For each item, the field name will be the value of field 'PropName' - // The field terms will be derived from field 'PropValue' + $this->storeAllFields(FieldStorage::no()); - $this->map = - "docs.Products.Select(p => new { " . - " _ = p.attributes.Select(item => this.CreateField(item.propName, item.propValue)), " . - " Name = p.name " . - "})"; + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + $this->analyze(DocumentsIndexingFields::ALL_FIELDS, "StopAnalyzer"); } } -`} - +``` - - -{`class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask + +```php +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { public function __construct() { parent::__construct(); $this->setMaps([ - "map('Products', function (p) { + "map('Products', function (product) { return { - _: p.Attributes.map(item => createField(item.PropName, item.PropValue, - { indexing: 'Search', storage: true, termVector: null })), - Name: p.Name + _: Object.keys(product.descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" ]); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + $fields = new IndexFieldOptionsArray(); + $allFieldsOptions = new IndexFieldOptions(); + $allFieldsOptions->setStorage(FieldStorage::no()); + $allFieldsOptions->setAnalyzer("StopAnalyzer"); + $fields->offsetSet(DocumentsIndexingFields::ALL_FIELDS, $allFieldsOptions); + $this->setFields($fields); } } -`} - - - - -* **The query**: - To get all documents matching a specific attribute property use: - - - -{`/** @var array $matchingDocuments */ -$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Attributes_ByName::class) - // 'Width' is a dynamic-index-field that was indexed from the Attributes list - ->whereEquals("Width", 10) - ->toList(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the Attributes list -from index 'Attributes/ByName' where Width = 10 -`} - +``` + + + + +### Which analyzer is used for the query term? +The following applies when a `search()` query targets a **dynamic field**: -## CreateField syntax +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `$this->analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `$this->analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | - - -{`object CreateField(string name, object value); + + + + + + +#### Syntax for LINQ-index: + + +```php +object CreateField(string name, object value); object CreateField(string name, object value, bool stored, bool analyzed); object CreateField(string name, object value, CreateFieldOptions options); -`} - +``` -| Parameters | Type | Description | -|------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **name** | `string` | Name of the dynamic-index-field | -| **value** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`False` - will set `FieldStorage.NO` (default value)
`True` - will set `FieldStorate.YES` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`None` - `FieldIndexing.Default` (default value)
`False` - `FieldIndexing.Exact`
`True` - `FieldIndexing.Search` | -| **options** | `CreateFieldOptions` | Dynamic-index-field options | +#### Syntax for JavaScript-index: -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| -| **Storage** | `FieldStorage` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing` | | -| **TermVector** | `FieldTermVector` | | + +```js +createField(fieldName, fieldValue, options); // returns object +``` + + +| Parameters | Type | Description | +|----------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage::no()` (default value)
`true` - will set `FieldStorage::yes()` | +| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing::default()` (default value)
`false` - `FieldIndexing::exact()`
`true` - `FieldIndexing::search()` | +| **options** | `CreateFieldOptions` | Dynamic-index-field options | + +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| +| **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -544,17 +976,15 @@ object CreateField(string name, object value, CreateFieldOptions options); + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. - +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-python.mdx b/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-python.mdx index 69514a29e2..6862d18067 100644 --- a/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-python.mdx +++ b/versioned_docs/version-6.2/indexes/content/_using-dynamic-fields-python.mdx @@ -1,490 +1,886 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. - -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. - -* In this page: - - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -## Indexing documents fields KEYS - -## Example - index any field under object + - - -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + + +### Example - index every field under an object + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. * **The document**: - - -{`class Product: - def __init__(self, Id: str = None, attributes: Dict[str, object] = None): - self.Id = Id - - # The KEYS under the Attributes object will be dynamically indexed - # Fields added to this object after index creation time will also get indexed - self.attributes = attributes -`} - - - - -{`// Sample document content -\{ - "Attributes": \{ - "Color": "Red", - "Size": 42 - \} -\} -`} - - + + + ```python + class Product: + def __init__(self, Id: str = None, Attributes: Dict[str, object] = None): + self.Id = Id + + # The KEYS under the Attributes object will be dynamically indexed + # Fields added to this object after index creation time will also get indexed + self.Attributes = Attributes + ``` + + + + ```json + // Sample document content + { + "Attributes": { + "Color": "Red", + "Size": 42 + } + } + ``` + * **The index**: - The below index will index any field under the `Attributes` object from the document, + + The following index will index any field under the `Attributes` object from the document, a dynamic-index-field will be created for each such field. New fields added to the object after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAttributeKey(AbstractIndexCreationTask): - def __init__(self): - super().__init__() - self.map = ( - "from p in docs.Products select new {" - "_ = p.attributes.Select(item => CreateField(item.Key, item.Value))" - "}" - ) -`} - - - - -{`class Products_ByAttributeKey_JS(AbstractJavaScriptIndexCreationTask): - def __init__(self): - super().__init__() - self.maps = { - """ - map('Products', function (p) { - return { - _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], - { indexing: 'Search', storage: true, termVector: null })) - }; - }) - """ - } -`} - - - - + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. + + + + ```python + class Products_ByAttributeKey(AbstractIndexCreationTask): + def __init__(self): + super().__init__() + self.map = ( + "from p in docs.Products select new { " + # Call 'CreateField' to generate dynamic-index-fields + # from the Attributes object keys. + + # Using '_' is just a convention. + # Any other string can be used instead of '_' + + # The actual field name will be 'item.Key' + # The actual field terms will be derived from 'item.Value' + "_ = p.Attributes.Select(item => CreateField(item.Key, item.Value)) " + "}" + ) + ``` + + + ```python + class Products_ByAttributeKey_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (p) { + return { + // The field name will be the key + // The field terms will be derived from the corresponding value + _: Object.keys(p.Attributes) + .map(key => createField(key, p.Attributes[key])) + }; + }) + """ + ] + ``` + + * **The query**: - * You can now query the generated dynamic-index fields. - * To get all documents with some 'size' use: - - - -{`matching_documents = list( - session.query_index_type(Products_ByAttributeKey, Product) - # 'size' is a dynamic-index-field that was indexed from the attributes object - .where_equals("size", 42) -) -`} - - - - -{`// 'Size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey' where Size = 42 -`} - - - - -## Example - index any field - - + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _Size_ use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Products_ByAttributeKey, Product) + # 'Size' is a dynamic-index-field that was indexed from the Attributes object + .where_equals("Size", 42) + ) + ``` + + + ```sql + // 'Size' is a dynamic-index-field that was indexed from the Attributes object + from index 'Products/ByAttributeKey' where Size = 42 + ``` + + + + + Dynamic index fields are queried with _document_query_ or _RQL_ because the field names (e.g. `Size`, `Color`) + are generated at indexing time and are not properties on the _Product_ class. + + Use the string-based `where_equals("Size", 42)` or RQL `where Size = 42`. + + + + + + +### Example - index every field + +The following example allows you to: * Define an index on a collection **without** needing any common structure between the indexed documents. - * After index is deployed, any new field added to the document will be indexed as well. - + * After the index is deployed, any new field added to any document in the collection will be indexed as well. - + -Consider whether this is really necessary, as indexing every single field can end up costing time and disk space. +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`class Product: - def __init__(self, Id: str = None, first_name: str = None, last_name: str = None, title: str = None): - self.Id = Id - - # All KEYS in the document will be dynamically indexes - # Fields added to the document after index creation time wil also get indexed - self.first_name = first_name - self.last_name = last_name - self.title = title - # ... -`} - - - - - -{`// Sample document content - \{ + + + ```python + class Product: + def __init__( + self, + Id: str = None, + FirstName: str = None, + LastName: str = None, + Title: str = None, + ): + self.Id = Id + + # All KEYS in the document will be dynamically indexed + # Fields added to the document after index creation time will also get indexed + self.FirstName = FirstName + self.LastName = LastName + self.Title = Title + # ... + ``` + + + + ```json + // Sample document content + { "FirstName": "John", "LastName": "Doe", "Title": "Engineer", // ... -\} -`} - - + } + ``` + * **The index**: - The below index will index any field from the document, + + The following index will index any field from the document, a dynamic-index-field will be created for each field. New fields added to the document after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAnyField_JS(AbstractJavaScriptIndexCreationTask): - def __init__(self): - super().__init__() - # This will index EVERY FIELD under the top level of the document - self.maps = { - """ - map('Products', function (p) { - return { - _: Object.keys(p).map(key => createField(key, p[key], - { indexing: 'Search', storage: true, termVector: null })) - } - }) - """ - } -`} - - - + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. + + The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection. + + + + ```python + class Products_ByAnyField_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + # This will index EVERY FIELD under the top level of the document + self.maps = [ + """ + map('Products', function (p) { + return { + _: Object.keys(p).map(key => createField(key, p[key])) + }; + }) + """ + ] + ``` + + * **The query**: - * To get all documents with some 'LastName' use: - - - -{`# 'last_name' is a dynamic-index-field that was indexed from the document -matching_documents = list( - session.query_index_type(Products_ByAnyField_JS, Product).where_equals("last_name", "Doe") -) -`} - - - - -{`// 'LastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where LastName = "Doe" -`} - - - + + To get all documents with a specific _LastName_ use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Products_ByAnyField_JS, Product) + # 'LastName' is a dynamic-index-field that was indexed from the document + .where_equals("LastName", "Doe") + ) + ``` + + + ```sql + // 'LastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where LastName = "Doe" + ``` + + + + + + + + + + + +### Example - basic + + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + + +* **The document**: + + + ```python + class Product: + def __init__(self, Id: str = None, ProductType: str = None, PricePerUnit: int = None): + self.Id = Id + + # The VALUE of ProductType will be dynamically indexed + self.ProductType = ProductType + self.PricePerUnit = PricePerUnit + ``` + + + + ```json + // Sample document content + { + "ProductType": "Electronics", + "PricePerUnit": 23 + } + ``` + + +* **The index**: + + The following index will index the **value** of document field 'ProductType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + + ```python + class Products_ByProductType(AbstractIndexCreationTask): + def __init__(self): + super().__init__() + self.map = ( + "from p in docs.Products select new { " + # Call 'CreateField' to generate the dynamic-index-fields + # The field name will be the value of document field 'ProductType' + # The field terms will be derived from document field 'PricePerUnit' + "_ = CreateField(p.ProductType, p.PricePerUnit) " + "}" + ) + ``` + + + ```python + class Products_ByProductType_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (p) { + return { + // The field name will be the value of document field 'ProductType' + // The field terms will be derived from document field 'PricePerUnit' + _: createField(p.ProductType, p.PricePerUnit) + }; + }) + """ + ] + ``` + + + +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Products_ByProductType, Product) + # 'Electronics' is the dynamic-index-field that was indexed + # from document field 'ProductType' + .where_equals("Electronics", 23) + ) + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'ProductType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + + +### Example - list + + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. + + +* **The document**: + + + ```python + class Attribute: + def __init__(self, PropName: str = None, PropValue: str = None): + self.PropName = PropName + self.PropValue = PropValue + + + class Product: + def __init__(self, Id: str = None, Name: str = None, Attributes: List[Attribute] = None): + self.Id = Id + self.Name = Name + + # For each element in this list, + # the VALUE of property 'PropName' will be dynamically indexed. + # e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + self.Attributes = Attributes + ``` + + + + ```json + // Sample document content + { + "Name": "SomeName", + "Attributes": [ + { + "PropName": "Color", + "PropValue": "Blue" + }, + { + "PropName": "Width", + "PropValue": "10" + }, + { + "PropName": "Length", + "PropValue": "20" + }, + // ... + ] + } + ``` + + +* **The index**: + + The following index will create a dynamic-index-field per item in the document's `Attributes` list. + New items added to the Attributes list after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the item's PropName **value**. + E.g., 'PropName' value `Width` will be a dynamic-index-field. + + + + ```python + class Attributes_ByName(AbstractIndexCreationTask): + def __init__(self): + super().__init__() + self.map = ( + "from a in docs.Products select new { " + # Define the dynamic-index-fields by calling 'CreateField' + # A dynamic-index-field will be generated for each item in 'Attributes' + + # For each item, the field name will be the value of field 'PropName' + # The field terms will be derived from field 'PropValue' + "_ = a.Attributes.Select(item => CreateField(item.PropName, item.PropValue)), " + + # A regular index field can be defined as well: + "Name = a.Name " + "}" + ) + ``` + + + ```python + class Attributes_ByName_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (p) { + return { + // For each item, + // the field name will be the value of field 'PropName' + // The field terms will be derived from field 'PropValue' + _: p.Attributes.map(item => createField(item.PropName, item.PropValue)), + + // A regular index field can be defined as well: + Name: p.Name + }; + }) + """ + ] + ``` + + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Attributes_ByName, Product) + # 'Width' is a dynamic-index-field that was indexed from the Attributes list + .where_equals("Width", 10) + ) + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the Attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `FieldIndexing.Search` in `CreateFieldOptions`. +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). -## Indexing documents fields VALUES +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. -## Example - basic + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. - + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```python + class Product: + def __init__(self, Id: str = None, Descriptions: Dict[str, str] = None): + self.Id = Id + self.Descriptions = Descriptions + ``` + + + + ```json + // Sample document content + { + "Descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + -* **The document**: - - -{`class Product: - def __init__(self, Id: str = None, product_type: str = None, price_per_unit: float = None): - self.Id = Id - - # The VALUE of ProductType will be dynamically indexed - self.product_type = product_type - self.price_per_unit = price_per_unit -`} - - +* **The index**: - - -{`// Sample document content -\{ - "ProductType": "Electronics", - "PricePerUnit": 23 -\} -`} - - + The following index creates one dynamic field for each entry in the `Descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. -* **The index**: - The below index will index the **value** of document field 'ProductType'. - - This value will be the dynamic-index-field name on which you can query. - E.g., Field value `Electronics` will be the dynamic-index-field. + Each generated dynamic field is indexed for **full-text search**. + + + + ```python + class Products_ByLocalizedDescription(AbstractIndexCreationTask): + USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = ( + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery" + ) + + def __init__(self): + super().__init__() + self.map = ( + "from product in docs.Products select new { " + " _ = product.Descriptions.Select(x => " + " CreateField(" + " \"Description_\" + x.Key, " + " x.Value, " + # Index each generated dynamic field for FTS + " new CreateFieldOptions { " + " Indexing = FieldIndexing.Search, " + " Storage = FieldStorage.No " + " })) " + "}" + ) + + self._store_all_fields(FieldStorage.NO) + + # Set this configuration key to true so that search() queries on dynamic fields + # will use the 'Default Search Analyzer' when the generated field name + # has no explicit analyzer configuration. + self.configuration = { + Products_ByLocalizedDescription.USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS: "true" + } + ``` + + + ```python + class Products_ByLocalizedDescription_JS(AbstractJavaScriptIndexCreationTask): + USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = ( + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery" + ) + + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (product) { + return { + _: Object.keys(product.Descriptions).map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: 'Search', + storage: 'No' + }); + }) + }; + }) + """ + ] + + # Apply a storage default to every generated field + self.fields = { + constants.Documents.Indexing.Fields.ALL_FIELDS: + IndexFieldOptions(storage=FieldStorage.NO) + } + + # Set this configuration key to true so that search() queries on dynamic fields + # will use the 'Default Search Analyzer' when the generated field name + # has no explicit analyzer configuration. + self.configuration = { + Products_ByLocalizedDescription_JS.USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS: "true" + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```python + results = list( + session.advanced + .document_query_from_index_type(Products_ByLocalizedDescription, Product) + .search("Description_English", "north wind") + ) + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + + +--- + + + +### Configuring analyzers for dynamic fields + +`CreateFieldOptions` does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field - -{`class Products_ByProductType(AbstractIndexCreationTask): +```python +class Products_ByLocalizedDescription(AbstractIndexCreationTask): def __init__(self): super().__init__() + self.map = ( + "from product in docs.Products select new { " + " _ = product.Descriptions.Select(x => " + " CreateField(" + " \"Description_\" + x.Key, " + " x.Value, " + " new CreateFieldOptions { " + " Indexing = FieldIndexing.Search, " + " Storage = FieldStorage.No " + " })) " + "}" + ) - # Call 'CreateField' to generate the dynamic-index-fields - # The field name will be the value of document field 'product_type' - # The field terms will be derived from document field 'price_per_unit' - self.map = "from p in docs.Products select new { _ = CreateField(p.product_type, p.price_per_unit)}" -`} - + self._store_all_fields(FieldStorage.NO) + + # Configure an analyzer for a known generated dynamic field name + # ============================================================== + self._analyze("Description_English", "StopAnalyzer") +``` - - -{`class Products_ByProductType_JS(AbstractJavaScriptIndexCreationTask): + +```python +class Products_ByLocalizedDescription_JS(AbstractJavaScriptIndexCreationTask): def __init__(self): super().__init__() - self.maps = { + self.maps = [ """ - map('Products', function (p) { + map('Products', function (product) { return { - _: createField(p.product_type, p.price_per_unit, - { indexing: 'Search', storage: true, termVector: null }) + _: Object.keys(product.Descriptions).map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; }) """ + ] + + # Configure an analyzer for a known generated dynamic field name + # ============================================================== + self.fields = { + constants.Documents.Indexing.Fields.ALL_FIELDS: + IndexFieldOptions(storage=FieldStorage.NO), + "Description_English": + IndexFieldOptions(analyzer="StopAnalyzer") } -`} - - - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`# 'electronics' is the dynamic-index-field that was indexed from the document 'product_type' -matching_documents = list( - session.advanced.document_query_from_index_type(Products_ByProductType, Product).where_equals( - "electronics", 23 - ) -) -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -from index 'Products/ByProductType' where Electronics = 23 -`} - +``` -## Example - list - - - -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. - - - -* **The document**: - - -{`class Attribute: - def __init__(self, prop_name: str = None, prop_value: str = None): - self.prop_name = prop_name - self.prop_value = prop_value - - -class Product: - def __init__(self, Id: str = None, name: str = None, attributes: List[Attribute] = None): - self.Id = Id - self.name = name - # For each element in this list, the VALUE of property 'prop_name' will be dynamically indexed - # e.g. color, width, length (in ex. below) will become dynamic-index-field - self.attributes = attributes -`} - - - - - -{`// Sample document content -\{ -Name": "SomeName", -Attributes": [ - \{ - "PropName": "Color", - "PropValue": "Blue" - \}, - \{ - "PropName": "Width", - "PropValue": "10" - \}, - \{ - "PropName": "Length", - "PropValue": "20" - \}, - ... - -\} -`} - - - -* **The index**: - The below index will create a dynamic-index-field per item in the document's `Attributes` list. - New items added to the Attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's PropName **value**. - E.g., 'PropName' value `width` will be a dynamic-index-field. +#### Configure a fallback analyzer for all fields - -{`class Attributes_ByName(AbstractIndexCreationTask): +```python +class Products_ByLocalizedDescription(AbstractIndexCreationTask): def __init__(self): super().__init__() self.map = ( - "from a in docs.Products select new " - "{ _ = a.attributes.Select( item => CreateField(item.prop_name, item.prop_value)), name = a.name " + "from product in docs.Products select new { " + " _ = product.Descriptions.Select(x => " + " CreateField(" + " \"Description_\" + x.Key, " + " x.Value, " + " new CreateFieldOptions { " + " Indexing = FieldIndexing.Search, " + " Storage = FieldStorage.No " + " })) " "}" ) -`} - + + self._store_all_fields(FieldStorage.NO) + + # Use this as a fallback for fields that do not have + # their own explicit analyzer configuration + # ================================================== + self._analyze(constants.Documents.Indexing.Fields.ALL_FIELDS, "StopAnalyzer") +``` - - -{`class Attributes_ByName_JS(AbstractJavaScriptIndexCreationTask): + +```python +class Products_ByLocalizedDescription_JS(AbstractJavaScriptIndexCreationTask): def __init__(self): super().__init__() - self.maps = { + self.maps = [ """ - map('Products', function (p) { + map('Products', function (product) { return { - _: p.Attributes.map(item => createField(item.PropName, item.PropValue, - { indexing: 'Search', storage: true, termVector: null })), - Name: p.Name + _: Object.keys(product.Descriptions).map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; }) """ + ] + + # Use this as a fallback for fields that do not have + # their own explicit analyzer configuration + # ================================================== + self.fields = { + constants.Documents.Indexing.Fields.ALL_FIELDS: + IndexFieldOptions(storage=FieldStorage.NO, analyzer="StopAnalyzer") } -`} - - - - -* **The query**: - To get all documents matching a specific attribute property use: - - - -{`matching_documents = list( - session.advanced.document_query_from_index_type(Attributes_ByName, Product).where_equals( - "width", 10 - ) -) -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the Attributes list -from index 'Attributes/ByName' where Width = 10 -`} - +``` + + + + +### Which analyzer is used for the query term? + +The following applies when a `search()` query targets a **dynamic field**: + +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `self._analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `self._analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | + + + -## CreateField syntax + #### Syntax for Index: - - -{`object CreateField(string name, object value); + +```python +CreateField(name, value) -object CreateField(string name, object value, bool stored, bool analyzed); +CreateField(name, value, stored, analyzed) -object CreateField(string name, object value, CreateFieldOptions options); -`} - +CreateField(name, value, CreateFieldOptions) +``` #### Syntax for JavaScript-index: - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` -| Parameters | Type | Description | -|------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **name** | `string` | Name of the dynamic-index-field | -| **value** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`False` - will set `FieldStorage.NO` (default value)
`True` - will set `FieldStorate.YES` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`None` - `FieldIndexing.Default` (default value)
`False` - `FieldIndexing.Exact`
`True` - `FieldIndexing.Search` | -| **options** | `CreateFieldOptions` | Dynamic-index-field options | +| Parameters | Type | Description | +|----------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorage.Yes` | +| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | +| **options** | `CreateFieldOptions` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| | **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -496,17 +892,15 @@ object CreateField(string name, object value, CreateFieldOptions options); + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/versioned_docs/version-6.2/indexes/querying/content/_searching-csharp.mdx b/versioned_docs/version-6.2/indexes/querying/content/_searching-csharp.mdx index d20ef5a65e..f0be0e6ffd 100644 --- a/versioned_docs/version-6.2/indexes/querying/content/_searching-csharp.mdx +++ b/versioned_docs/version-6.2/indexes/querying/content/_searching-csharp.mdx @@ -16,11 +16,13 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) * [Searching with wildcards](../../../indexes/querying/searching.mdx#searching-with-wildcards) * [When using RavenStandardAnalyzer or StandardAnalyzer or NGramAnalyzer](../../../indexes/querying/searching.mdx#when-usingoror) @@ -28,6 +30,7 @@ import CodeBlock from '@theme/CodeBlock'; * [When using the Exact analyzer](../../../indexes/querying/searching.mdx#when-using-the-exact-analyzer) + ## Indexing single field for FTS #### The index: @@ -262,7 +265,7 @@ where (search(EmployeeData, "Manager") or search(EmployeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method in the _Map_ function to extract all property values and index them in a single searchable field. * This approach makes the index robust to changes in the document schema. @@ -377,7 +380,14 @@ where search(AllValues, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). ## Boosting search results diff --git a/versioned_docs/version-6.2/indexes/querying/content/_searching-java.mdx b/versioned_docs/version-6.2/indexes/querying/content/_searching-java.mdx index 7d85246ced..4accf3f512 100644 --- a/versioned_docs/version-6.2/indexes/querying/content/_searching-java.mdx +++ b/versioned_docs/version-6.2/indexes/querying/content/_searching-java.mdx @@ -388,4 +388,11 @@ where search(allValues, "tofu") +## Indexing dynamic fields for FTS +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). \ No newline at end of file diff --git a/versioned_docs/version-6.2/indexes/querying/content/_searching-nodejs.mdx b/versioned_docs/version-6.2/indexes/querying/content/_searching-nodejs.mdx index 4ca66b5877..cb73bfd7c0 100644 --- a/versioned_docs/version-6.2/indexes/querying/content/_searching-nodejs.mdx +++ b/versioned_docs/version-6.2/indexes/querying/content/_searching-nodejs.mdx @@ -5,7 +5,7 @@ import CodeBlock from '@theme/CodeBlock'; -* Prior to reading this article, please refer to [full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) +* Prior to reading this article, please refer to [Full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) to learn about the `search` method. * **All capabilities** provided by `search` with a dynamic query can also be used when querying a static-index. @@ -17,11 +17,13 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) * [Searching with wildcards](../../../indexes/querying/searching.mdx#searching-with-wildcards) * [When using RavenStandardAnalyzer or StandardAnalyzer or NGramAnalyzer](../../../indexes/querying/searching.mdx#when-usingoror) @@ -29,6 +31,7 @@ import CodeBlock from '@theme/CodeBlock'; * [When using the Exact analyzer](../../../indexes/querying/searching.mdx#when-using-the-exact-analyzer) + ## Indexing single field for FTS #### The index: @@ -163,7 +166,7 @@ where (search(employeeData, "Manager") or search(employeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, which is available when using **a C# LINQ string** that is assigned to the `map` property in the Node.js index class, as shown in the example below. @@ -229,6 +232,14 @@ where search(AllValues, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). ## Boosting search results diff --git a/versioned_docs/version-6.2/indexes/querying/content/_searching-php.mdx b/versioned_docs/version-6.2/indexes/querying/content/_searching-php.mdx index 55261ce19f..510a6f8752 100644 --- a/versioned_docs/version-6.2/indexes/querying/content/_searching-php.mdx +++ b/versioned_docs/version-6.2/indexes/querying/content/_searching-php.mdx @@ -5,7 +5,7 @@ import CodeBlock from '@theme/CodeBlock'; -* Prior to reading this article, please refer to [full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) +* Prior to reading this article, please refer to [Full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) to learn about the `search` method. * **All capabilities** provided by `search` with a dynamic query can also be used when querying a static-index. @@ -17,14 +17,17 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) + ## Indexing single field for FTS #### The index: @@ -253,7 +256,7 @@ where (search(EmployeeData, "Manager") or search(EmployeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, which is available in the **C# LINQ string** that is assigned to the `map` property in the PHP index class, as shown in the example below. @@ -332,6 +335,15 @@ where search(allValues, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). + ## Boosting search results diff --git a/versioned_docs/version-6.2/indexes/querying/content/_searching-python.mdx b/versioned_docs/version-6.2/indexes/querying/content/_searching-python.mdx index f06c29d94f..88c6aa452f 100644 --- a/versioned_docs/version-6.2/indexes/querying/content/_searching-python.mdx +++ b/versioned_docs/version-6.2/indexes/querying/content/_searching-python.mdx @@ -5,7 +5,7 @@ import CodeBlock from '@theme/CodeBlock'; -* Prior to reading this article, please refer to [full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) +* Prior to reading this article, please refer to [Full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) to learn about the `search` method. * **All capabilities** provided by `search` with a dynamic query can also be used when querying a static-index. @@ -17,14 +17,17 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) + ## Indexing single field for FTS #### The index: @@ -167,7 +170,7 @@ where (search(EmployeeData, "Manager") or search(EmployeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, which is available in the **C# LINQ string** that is assigned to the `map` property in the Python index class, as shown in the example below. @@ -238,6 +241,14 @@ where search(all_values, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). ## Boosting search results diff --git a/versioned_docs/version-6.2/indexes/using-dynamic-fields.mdx b/versioned_docs/version-6.2/indexes/using-dynamic-fields.mdx index e9ffb33a07..e465ba5431 100644 --- a/versioned_docs/version-6.2/indexes/using-dynamic-fields.mdx +++ b/versioned_docs/version-6.2/indexes/using-dynamic-fields.mdx @@ -1,7 +1,8 @@ --- -title: "Indexes: Dynamic Index Fields" -sidebar_label: Dynamic Fields -sidebar_position: 28 +title: "Dynamic Index Fields" +sidebar_label: "Dynamic Index Fields" +description: "Index document fields dynamically in RavenDB when field names are unknown at index definition time, using the CreateField method." +sidebar_position: 27 supported_languages: ["csharp", "java", "python", "php", "nodejs"] see_also: - title: "Boosting" @@ -31,7 +32,6 @@ import UsingDynamicFieldsPython from './content/_using-dynamic-fields-python.mdx import UsingDynamicFieldsPhp from './content/_using-dynamic-fields-php.mdx'; import UsingDynamicFieldsNodejs from './content/_using-dynamic-fields-nodejs.mdx'; - diff --git a/versioned_docs/version-6.2/server/configuration/indexing-configuration.mdx b/versioned_docs/version-6.2/server/configuration/indexing-configuration.mdx index fe64090eb5..e845bf3326 100644 --- a/versioned_docs/version-6.2/server/configuration/indexing-configuration.mdx +++ b/versioned_docs/version-6.2/server/configuration/indexing-configuration.mdx @@ -95,6 +95,7 @@ import LanguageContent from "@site/src/components/LanguageContent"; [Indexing.OrderByTicksAutomaticallyWhenDatesAreInvolved](../../server/configuration/indexing-configuration.mdx#indexingorderbyticksautomaticallywhendatesareinvolved) [Indexing.QueryClauseCache.Disabled](../../server/configuration/indexing-configuration.mdx#indexingqueryclausecachedisabled) [Indexing.QueryClauseCache.RepeatedQueriesTimeFrameInSec](../../server/configuration/indexing-configuration.mdx#indexingqueryclausecacherepeatedqueriestimeframeinsec) + [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) [Indexing.ScratchSpaceLimitInMb](../../server/configuration/indexing-configuration.mdx#indexingscratchspacelimitinmb) [Indexing.Static.SearchEngineType](../../server/configuration/indexing-configuration.mdx#indexingstaticsearchenginetype) [Indexing.Throttling.TimeIntervalInMs](../../server/configuration/indexing-configuration.mdx#indexingthrottlingtimeintervalinms) @@ -893,6 +894,33 @@ Queries that repeat within this time frame will be considered worth caching. +## Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery + +* This setting controls which analyzer is used for the **query term** when a **`search()` query** targets a [Dynamic index field](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + that was indexed for full-text search, and no analyzer is explicitly configured for that field in the index definition. + + * `false` (default): + The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) is used to analyze the **query term**. + This analyzer can be configured via the [Indexing.Analyzers.Default](../../server/configuration/indexing-configuration.mdx#indexinganalyzersdefault) configuration key. + + * `true`: + The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) is used to analyze the **query term**. + This analyzer can be configured via the [Indexing.Analyzers.Search.Default](../../server/configuration/indexing-configuration.mdx#indexinganalyzerssearchdefault) configuration key. + +* See the table in [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + for the complete analyzer-selection rules based on this configuration key and the analyzer configured for the dynamic index field. + +* This configuration applies to both the Lucene and Corax search engines. + The default is `false` to preserve backward compatibility. + +--- + +- **Type**: `bool` +- **Default**: `false` +- **Scope**: Server-wide, or per database, or per index + + + ## Indexing.ScratchSpaceLimitInMb * Amount of scratch space in megabytes that we allow to use for the index storage. diff --git a/versioned_docs/version-7.0/indexes/assets/dynamic-index-fields-1.png b/versioned_docs/version-7.0/indexes/assets/dynamic-index-fields-1.png index b13d9076f8..a20adda0ec 100644 Binary files a/versioned_docs/version-7.0/indexes/assets/dynamic-index-fields-1.png and b/versioned_docs/version-7.0/indexes/assets/dynamic-index-fields-1.png differ diff --git a/versioned_docs/version-7.0/indexes/assets/dynamic-index-fields-2.png b/versioned_docs/version-7.0/indexes/assets/dynamic-index-fields-2.png index dd2b4082ea..c5b1bc14bc 100644 Binary files a/versioned_docs/version-7.0/indexes/assets/dynamic-index-fields-2.png and b/versioned_docs/version-7.0/indexes/assets/dynamic-index-fields-2.png differ diff --git a/versioned_docs/version-7.0/indexes/content/_using-analyzers-csharp.mdx b/versioned_docs/version-7.0/indexes/content/_using-analyzers-csharp.mdx index 6022cfd145..2e824fea4b 100644 --- a/versioned_docs/version-7.0/indexes/content/_using-analyzers-csharp.mdx +++ b/versioned_docs/version-7.0/indexes/content/_using-analyzers-csharp.mdx @@ -24,7 +24,7 @@ import CodeBlock from '@theme/CodeBlock'; * [Understanding the role of analyzers](../../indexes/using-analyzers.mdx#understanding-the-role-of-analyzers) * [Analyzers available in RavenDB](../../indexes/using-analyzers.mdx#analyzers-available-in-ravendb) * [Setting analyzer for index-field](../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field) - * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendb) + * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendbs-default-analyzers) * [Disabling indexing for index-field](../../indexes/using-analyzers.mdx#disabling-indexing-for-index-field) * [Creating custom analyzers](../../indexes/using-analyzers.mdx#creating-custom-analyzers) * [Viewing the indexed terms](../../indexes/using-analyzers.mdx#viewing-the-indexed-terms) diff --git a/versioned_docs/version-7.0/indexes/content/_using-analyzers-nodejs.mdx b/versioned_docs/version-7.0/indexes/content/_using-analyzers-nodejs.mdx index 6afd3b6202..fae2cdd2b9 100644 --- a/versioned_docs/version-7.0/indexes/content/_using-analyzers-nodejs.mdx +++ b/versioned_docs/version-7.0/indexes/content/_using-analyzers-nodejs.mdx @@ -24,7 +24,7 @@ import CodeBlock from '@theme/CodeBlock'; * [Understanding the role of analyzers](../../indexes/using-analyzers.mdx#understanding-the-role-of-analyzers) * [Analyzers available in RavenDB](../../indexes/using-analyzers.mdx#analyzers-available-in-ravendb) * [Setting analyzer for index-field](../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field) - * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendb) + * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendbs-default-analyzers) * [Disabling indexing for index-field](../../indexes/using-analyzers.mdx#disabling-indexing-for-index-field) * [Creating custom analyzers](../../indexes/using-analyzers.mdx#creating-custom-analyzers) * [Viewing the indexed terms](../../indexes/using-analyzers.mdx#viewing-the-indexed-terms) diff --git a/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-csharp.mdx b/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-csharp.mdx index b4c4f518df..5bf052d482 100644 --- a/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-csharp.mdx +++ b/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-csharp.mdx @@ -2,533 +2,923 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. - -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. - -* In this page: - - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -## Indexing documents fields KEYS - -## Example - index any field under object - - + -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + + +### Example - index every field under an object + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. * **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - - // The KEYS under the Attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - public Dictionary Attributes \{ get; set; \} -\} -`} - - - - -{`// Sample document content -\{ - "Attributes": \{ - "Color": "Red", - "Size": 42 - \} -\} -`} - - + + + ```csharp + public class Product + { + public string Id { get; set; } + + // The KEYS under the Attributes object will be dynamically indexed + // Fields added to this object after index creation time will also get indexed + public Dictionary Attributes { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "Attributes": { + "Color": "Red", + "Size": 42 + } + } + ``` + * **The index**: - The below index will index any field under the `Attributes` object from the document, + + The following index will index any field under the `Attributes` object from the document, a dynamic-index-field will be created for each such field. New fields added to the object after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. - - - -{`public class Products_ByAttributeKey : AbstractIndexCreationTask -{ - public Products_ByAttributeKey() + + + ```csharp + public class Products_ByAttributeKey : AbstractIndexCreationTask { - Map = products => from p in products - select new - { - // Call 'CreateField' to generate dynamic-index-fields from the Attributes object keys - // Using '_' is just a convention. Any other string can be used instead of '_' - - // The actual field name will be item.Key - // The actual field terms will be derived from item.Value - _ = p.Attributes.Select(item => CreateField(item.Key, item.Value)) - }; + public Products_ByAttributeKey() + { + Map = products => from p in products + select new + { + // Call 'CreateField' to generate dynamic-index-fields + // from the Attributes object keys. + + // Using '_' is just a convention. + // Any other string can be used instead of '_' + + // The actual field name will be 'item.Key' + // The actual field terms will be derived from 'item.Value' + _ = p.Attributes.Select(item => CreateField(item.Key, item.Value)) + }; + } } -} -`} - - - - -{`public class Products_ByAttributeKey_JS : AbstractJavaScriptIndexCreationTask -{ - public Products_ByAttributeKey_JS() + ``` + + + ```csharp + public class Products_ByAttributeKey_JS : AbstractJavaScriptIndexCreationTask { - Maps = new HashSet + public Products_ByAttributeKey_JS() { - @"map('Products', function (p) { - return { - _: Object.keys(p.Attributes).map(key => createField(key, p.Attributes[key], - { indexing: 'Search', storage: true, termVector: null })) - }; - })" - }; + Maps = new HashSet + { + @"map('Products', function (p) { + return { + // The field name will be the key + // The field terms will be derived from the corresponding value + _: Object.keys(p.Attributes) + .map(key => createField(key, p.Attributes[key])) + }; + })" + }; + } } -} -`} - - - - + ``` + + * **The query**: - * You can now query the generated dynamic-index fields. - The `_` property is Not queryable but used only in the index definition syntax. - * To get all documents with some 'Size' use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'Size' is a dynamic-index-field that was indexed from the Attributes object - .WhereEquals("Size", 42) - .ToList(); -`} - - - - -{`// 'Size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey' where Size = 42 -`} - - - + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _Size_ use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'Size' is a dynamic-index-field that was indexed from the Attributes object + .WhereEquals("Size", 42) + .ToList(); + ``` + + + ```sql + // 'Size' is a dynamic-index-field that was indexed from the Attributes object + from index 'Products/ByAttributeKey' where Size = 42 + ``` + + + + + Dynamic index fields are queried with _DocumentQuery_ or _RQL_ because the field names (e.g. `Size`, `Color`) + are generated at indexing time and are not properties on the _Product_ class. + + A strongly typed LINQ query such as `Query().Where(p => p.Size == 42)` would not compile. + Use the string-based `WhereEquals("Size", 42)` or RQL `where Size = 42` instead. + -## Example - index any field + - + + +### Example - index every field + +The following example allows you to: * Define an index on a collection **without** needing any common structure between the indexed documents. - * After index is deployed, any new field added to the document will be indexed as well. - + * After the index is deployed, any new field added to any document in the collection will be indexed as well. - + -Consider whether this is really necessary, as indexing every single field can end up costing time and disk space. +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - public string FirstName \{ get; set; \} - public string LastName \{ get; set; \} - public string Title \{ get; set; \} - // ... -\} -`} - - - - - -{`// Sample document content - \{ - "FirstName": "John", - "LastName": "Doe", - "Title": "Engineer", + + + ```csharp + public class Product + { + public string Id { get; set; } + + // All KEYS in the document will be dynamically indexed + // Fields added to the document after index creation time will also get indexed + public string FirstName { get; set; } + public string LastName { get; set; } + public string Title { get; set; } // ... -\} -`} - - + } + ``` + + + + ```json + // Sample document content + { + "FirstName": "John", + "LastName": "Doe", + "Title": "Engineer", + // ... + } + ``` + * **The index**: - The below index will index any field from the document, + + The following index will index any field from the document, a dynamic-index-field will be created for each field. New fields added to the document after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. + + The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection. - - - -{`public class Products_ByAnyField_JS : AbstractJavaScriptIndexCreationTask -{ - public Products_ByAnyField_JS() + + + ```csharp + public class Products_ByAnyField_JS : AbstractJavaScriptIndexCreationTask { - // This will index EVERY FIELD under the top level of the document - Maps = new HashSet + public Products_ByAnyField_JS() { - @"map('Products', function (p) { - return { - _: Object.keys(p).map(key => createField(key, p[key], - { indexing: 'Search', storage: true, termVector: null })) - } - })" - }; + // This will index EVERY FIELD under the top level of the document + Maps = new HashSet + { + @"map('Products', function (p) { + return { + _: Object.keys(p).map(key => createField(key, p[key])) + }; + })" + }; + } } -} -`} - - - + ``` + + * **The query**: - * To get all documents with some 'LastName' use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'LastName' is a dynamic-index-field that was indexed from the document - .WhereEquals("LastName", "Doe") - .ToList(); -`} - - - - -{`// 'LastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where LastName = "Doe" -`} - - - + + To get all documents with a specific _LastName_ use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'LastName' is a dynamic-index-field that was indexed from the document + .WhereEquals("LastName", "Doe") + .ToList(); + ``` + + + ```sql + // 'LastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where LastName = "Doe" + ``` + + + + + + + + + +### Example - basic -## Indexing documents fields VALUES + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + -## Example - basic +* **The document**: + + + ```csharp + public class Product + { + public string Id { get; set; } + + // The VALUE of ProductType will be dynamically indexed + public string ProductType { get; set; } + public int PricePerUnit { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "ProductType": "Electronics", + "PricePerUnit": 23 + } + ``` + - +* **The index**: + + The following index will index the **value** of document field 'ProductType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + + ```csharp + public class Products_ByProductType : AbstractIndexCreationTask + { + public Products_ByProductType() + { + Map = products => from p in products + select new + { + // Call 'CreateField' to generate the dynamic-index-fields + // The field name will be the value of document field 'ProductType' + // The field terms will be derived from document field 'PricePerUnit' + _ = CreateField(p.ProductType, p.PricePerUnit) + }; + } + } + ``` + + + ```csharp + public class Products_ByProductType_JS : AbstractJavaScriptIndexCreationTask + { + public Products_ByProductType_JS() + { + Maps = new HashSet + { + @"map('Products', function (p) { + return { + // The field name will be the value of document field 'ProductType' + // The field terms will be derived from document field 'PricePerUnit' + _: createField(p.ProductType, p.PricePerUnit) + }; + })" + }; + } + } + ``` + + + +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'ProductType' + .WhereEquals("Electronics", 23) + .ToList(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'ProductType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. -* For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---index-a-list-of-properties) below. +### Example - list + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. * **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - - // The VALUE of ProductType will be dynamically indexed - public string ProductType \{ get; set; \} - public int PricePerUnit \{ get; set; \} -\} -`} - - - - - -{`// Sample document content -\{ - "ProductType": "Electronics", - "PricePerUnit": 23 -\} -`} - - + + + ```csharp + public class Product + { + public string Id { get; set; } + public string Name { get; set; } + + // For each element in this list, + // the VALUE of property 'PropName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + public List Attributes { get; set; } + } + + public class Attribute + { + public string PropName { get; set; } + public string PropValue { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "Name": "SomeName", + "Attributes": [ + { + "PropName": "Color", + "PropValue": "Blue" + }, + { + "PropName": "Width", + "PropValue": "10" + }, + { + "PropName": "Length", + "PropValue": "20" + }, + ... + ] + } + ``` + * **The index**: - The below index will index the **value** of document field 'ProductType'. + + The following index will create a dynamic-index-field per item in the document's `Attributes` list. + New items added to the Attributes list after index creation time will be dynamically indexed as well. - This value will be the dynamic-index-field name on which you can query. - E.g., Field value `Electronics` will be the dynamic-index-field. + The actual dynamic-index-field name on which you can query will be the item's PropName **value**. + E.g., 'PropName' value `Width` will be a dynamic-index-field. + + + + ```csharp + public class Attributes_ByName : AbstractIndexCreationTask + { + public Attributes_ByName() + { + Map = products => from a in products + select new + { + // Define the dynamic-index-fields by calling 'CreateField' + // A dynamic-index-field will be generated for each item in 'Attributes' + + // For each item, the field name will be the value of field 'PropName' + // The field terms will be derived from field 'PropValue' + _ = a.Attributes.Select(item => + CreateField(item.PropName, item.PropValue)), + + // A regular index field can be defined as well: + Name = a.Name + }; + } + } + ``` + + + ```csharp + public class Attributes_ByName_JS : AbstractJavaScriptIndexCreationTask + { + public Attributes_ByName_JS() + { + Maps = new HashSet + { + @"map('Products', function (p) { + return { + // For each item, + // the field name will be the value of field 'PropName' + // The field terms will be derived from field 'PropValue' + _: p.Attributes.map(item => createField( + item.PropName, item.PropValue)), + + // A regular index field can be defined as well: + Name: p.Name + }; + })" + }; + } + } + ``` + + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'Width' is a dynamic-index-field that was indexed from the Attributes list + .WhereEquals("Width", 10) + .ToList(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the Attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `FieldIndexing.Search` in `CreateFieldOptions`. + +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `AllFields` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `AllFields` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. + + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. + + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + + + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```csharp + public class Product + { + public string Id { get; set; } + + public Dictionary Descriptions { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "Descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `Descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. + + Each generated dynamic field is indexed for **full-text search**. + + + + ```csharp + public class Products_ByLocalizedDescription : AbstractIndexCreationTask + { + private const string UseSearchAnalyzerForDynamicFields = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription() + { + Map = products => from product in products + select new + { + _ = product.Descriptions.Select(x => + CreateField( + "Description_" + x.Key, + x.Value, + new CreateFieldOptions + { + // Index each generated dynamic field for FTS + Indexing = FieldIndexing.Search, + Storage = FieldStorage.No + })) + }; + + StoreAllFields(FieldStorage.No); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + Configuration[UseSearchAnalyzerForDynamicFields] = "true"; + } + } + ``` + + + ```csharp + public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask + { + private const string UseSearchAnalyzerForDynamicFields = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription_JS() + { + Maps = new HashSet + { + @"map('Products', function (product) { + return { + _: Object.keys(product.Descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: 'Search', + storage: 'No' + }); + }) + }; + })" + }; + + Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions + { + Storage = FieldStorage.No + }; + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + Configuration[UseSearchAnalyzerForDynamicFields] = "true"; + } + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```csharp + List results = session.Advanced + .DocumentQuery() + .Search("Description_English", "north wind") + .ToList(); + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + + +--- + + + +### Configuring analyzers for dynamic fields - - - -{`public class Products_ByProductType : AbstractIndexCreationTask +`CreateFieldOptions` does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `AllFields` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `AllFields` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field + + + +```csharp +public class Products_ByLocalizedDescription : AbstractIndexCreationTask { - public Products_ByProductType() + public Products_ByLocalizedDescription() { - Map = products => from p in products + Map = products => from product in products select new { - // Call 'CreateField' to generate the dynamic-index-fields - // The field name will be the value of document field 'ProductType' - // The field terms will be derived from document field 'PricePerUnit' - _ = CreateField(p.ProductType, p.PricePerUnit) + _ = product.Descriptions.Select(x => + CreateField( + "Description_" + x.Key, + x.Value, + new CreateFieldOptions + { + Indexing = FieldIndexing.Search, + Storage = FieldStorage.No + })) }; + + StoreAllFields(FieldStorage.No); + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + Analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -{`public class Products_ByProductType_JS : AbstractJavaScriptIndexCreationTask + +```csharp +public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask { - public Products_ByProductType_JS() + public Products_ByLocalizedDescription_JS() { Maps = new HashSet { - @"map('Products', function (p) { + @"map('Products', function (product) { return { - _: createField(p.ProductType, p.PricePerUnit, - { indexing: 'Search', storage: true, termVector: null }) + _: Object.keys(product.Descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" }; + + Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions + { + Storage = FieldStorage.No + }; + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + Fields["Description_English"] = new IndexFieldOptions + { + Analyzer = "StopAnalyzer" + }; } } -`} - - - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' - .WhereEquals("Electronics", 23) - .ToList(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -from index 'Products/ByProductType' where Electronics = 23 -`} - +``` -## Example - list - - +#### Configure a fallback analyzer for all fields -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. - - - -* **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - public string Name \{ get; set; \} - - // For each element in this list, the VALUE of property 'PropName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - public List Attributes \{ get; set; \} -\} - -public class Attribute -\{ - public string PropName \{ get; set; \} - public string PropValue \{ get; set; \} -\} -`} - - - - - -{`// Sample document content -\{ -Name": "SomeName", -Attributes": [ - \{ - "PropName": "Color", - "PropValue": "Blue" - \}, - \{ - "PropName": "Width", - "PropValue": "10" - \}, - \{ - "PropName": "Length", - "PropValue": "20" - \}, - ... - -\} -`} - - - -* **The index**: - The below index will create a dynamic-index-field per item in the document's `Attributes` list. - New items added to the Attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's PropName **value**. - E.g., 'PropName' value `Width` will be a dynamic-index-field. - - - - -{`public class Attributes_ByName : AbstractIndexCreationTask + + +```csharp +public class Products_ByLocalizedDescription : AbstractIndexCreationTask { - public Attributes_ByName() + public Products_ByLocalizedDescription() { - Map = products => from a in products + Map = products => from product in products select new { - // Define the dynamic-index-fields by calling CreateField - // A dynamic-index-field will be generated for each item in the Attributes list - - // For each item, the field name will be the value of field 'PropName' - // The field terms will be derived from field 'PropValue' - _ = a.Attributes.Select(item => CreateField(item.PropName, item.PropValue)), - - // A regular index field can be defined as well: - Name = a.Name + _ = product.Descriptions.Select(x => + CreateField( + "Description_" + x.Key, + x.Value, + new CreateFieldOptions + { + Indexing = FieldIndexing.Search, + Storage = FieldStorage.No + })) }; + + StoreAllFields(FieldStorage.No); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + Analyze(Constants.Documents.Indexing.Fields.AllFields, "StopAnalyzer"); } } -`} - +``` - - -{`public class Attributes_ByName_JS : AbstractJavaScriptIndexCreationTask + +```csharp +public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask { - public Attributes_ByName_JS() + public Products_ByLocalizedDescription_JS() { Maps = new HashSet { - @"map('Products', function (p) { + @"map('Products', function (product) { return { - _: p.Attributes.map(item => createField(item.PropName, item.PropValue, - { indexing: 'Search', storage: true, termVector: null })), - Name: p.Name + _: Object.keys(product.Descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" }; + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions + { + Storage = FieldStorage.No, + Analyzer = "StopAnalyzer" + }; } } -`} - - - - -* **The query**: - * To get all documents matching a specific attribute property use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'Width' is a dynamic-index-field that was indexed from the Attributes list - .WhereEquals("Width", 10) - .ToList(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the Attributes list -from index 'Attributes/ByName' where Width = 10 -`} - +``` + + + + +### Which analyzer is used for the query term? + +The following applies when a `search()` query targets a **dynamic field**: + +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `AllFields` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `AllFields` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `Analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `Analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | + -## CreateField syntax + + + #### Syntax for LINQ-index: - - -{`object CreateField(string name, object value); + +```csharp +object CreateField(string name, object value); object CreateField(string name, object value, bool stored, bool analyzed); object CreateField(string name, object value, CreateFieldOptions options); -`} - +``` #### Syntax for JavaScript-index: - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` | Parameters | Type | Description | |----------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| | **fieldName** | `string` | Name of the dynamic-index-field | -| **fieldValue** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorate.Yes` | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorage.Yes` | | **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | | **options** | `CreateFieldOptions` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| | **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -540,11 +930,15 @@ object CreateField(string name, object value, CreateFieldOptions options); -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. + + + +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) + + \ No newline at end of file diff --git a/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-java.mdx b/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-java.mdx index 17f3cbe360..d54ef6bea8 100644 --- a/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-java.mdx +++ b/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-java.mdx @@ -2,479 +2,918 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -* For such dynamic data, you can define indexes with **dynamic-index-fields**. + -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. + -* Any value type can be indexed, string, number, date, etc. + + +### Example - index every field under an object -* An index definition can contain both dynamic-index-fields and regular-index-fields. + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. + -* In this page: +* **The document**: + + + ```java + public class Product { + private String id; - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) + // The KEYS under the attributes object will be dynamically indexed + // Fields added to this object after index creation time will also get indexed + private Map attributes; - + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "attributes": { + "color": "Red", + "size": 42 + } + } + ``` + -## Indexing documents fields KEYS +* **The index**: + + The following index will index any field under the `attributes` object from the document, + a dynamic-index-field will be created for each such field. + New fields added to the object after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `color` & `size` will become the actual dynamic-index-fields. + + + + ```java + public class Products_ByAttributeKey extends AbstractIndexCreationTask { + public Products_ByAttributeKey() { + + // Call 'CreateField' to generate dynamic-index-fields + // from the attributes object keys. + + // Using '_' is just a convention. + // Any other string can be used instead of '_'. + + // The actual field name will be 'item.Key' + // The actual field terms will be derived from 'item.Value' + map = "docs.Products.Select(p => new { " + + " _ = p.attributes.Select(item => " + + " this.CreateField(item.Key, item.Value)) " + + "})"; + } + } + ``` + + + ```java + public class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByAttributeKey_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + // The field name will be the key + // The field terms will be derived from the corresponding value + " _: Object.keys(p.attributes) " + + " .map(key => createField(key, p.attributes[key])) " + + " }; " + + "})" + )); + } + } + ``` + + -## Example - index any field under object +* **The query**: + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _size_ use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Products_ByAttributeKey.class) + // 'size' is a dynamic-index-field that was indexed from the attributes object + .whereEquals("size", 42) + .toList(); + ``` + + + ```sql + // 'size' is a dynamic-index-field that was indexed from the attributes object + from index 'Products/ByAttributeKey' where size = 42 + ``` + + + + + Dynamic index fields are queried with _DocumentQuery_ or _RQL_ because the field names (e.g. `size`, `color`) + are generated at indexing time and are not properties on the _Product_ class. + + Use the string-based `whereEquals("size", 42)` or RQL `where size = 42` instead. + - + -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + + +### Example - index every field + +The following example allows you to: + * Define an index on a collection **without** needing any common structure between the indexed documents. + * After the index is deployed, any new field added to any document in the collection will be indexed as well. + + + +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`public class Product \{ - private String id; - - // The KEYS under the attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - private Dictionary attributes; - - // get + set implementation ... -\} -`} - - - - -{`// Sample document content -\{ - "attributes": \{ - "color": "Red", - "size": 42 - \} -\} -`} - - + + + ```java + public class Product { + private String id; + + // All KEYS in the document will be dynamically indexed + // Fields added to the document after index creation time will also get indexed + private String firstName; + private String lastName; + private String title; + // ... + + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "firstName": "John", + "lastName": "Doe", + "title": "Engineer" + // ... + } + ``` + * **The index**: - The following index will index any field under the `attributes` object from the document, - a dynamic-index-field will be created for each such field. - New fields added to the object after index creation time will be dynamically indexed as well. + + The following index will index any field from the document, + a dynamic-index-field will be created for each field. + New fields added to the document after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + + The example uses an `AbstractJavaScriptIndexCreationTask` because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a map or collection. + + + + ```java + public class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByAnyField_JS() { + + // This will index EVERY FIELD under the top level of the document + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + " _: Object.keys(p).map(key => createField(key, p[key])) " + + " }; " + + "})" + )); + } + } + ``` + + - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - e.g. Keys `color` & `size` will become the actual dynamic-index-fields. +* **The query**: + + To get all documents with a specific _lastName_ use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Products_ByAnyField_JS.class) + // 'lastName' is a dynamic-index-field that was indexed from the document + .whereEquals("lastName", "Doe") + .toList(); + ``` + + + ```sql + // 'lastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where lastName = "Doe" + ``` + + + + + + - - - -{`public class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { - public Products_ByAttributeKey_JS() { - // Call 'createField' to generate dynamic-index-fields from the attributes object keys - // Using '_' is just a convention. Any other string can be used instead of '_' + - // The actual field name will be the key - // The actual field terms will be derived from p.attributes[key] - setMaps(Sets.newHashSet( - "map('Products', function (p) { " + - " return { " + - " _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], " + - " { indexing: 'Search', storage: false, termVector: null })) " + - " }; " + - "}) " - )); - } -} -`} - - - + + +### Example - basic -* **The query**: - * You can now query the generated dynamic-index fields. - Property `_` is Not queryable, it is only used in the index definition syntax. - * To get all documents with some 'size' use: - - - -{`List matchingDocuments = session - .query(Product.class, Products_ByAttributeKey_JS.class) - .whereEquals("size", 42) - .toList(); -`} - - - - -{`// 'size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey/JS' where size = 42 -`} - - - + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + -## Example - index any field +* **The document**: + + + ```java + public class Product { + private String id; - + // The VALUE of productType will be dynamically indexed + private String productType; + private int pricePerUnit; -* Define an index on a collection **without** needing any common structure between the indexed documents. -* After index is deployed, any new field added to the document will be indexed as well. + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "productType": "Electronics", + "pricePerUnit": 23 + } + ``` + - +* **The index**: + + The following index will index the **value** of document field 'productType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + + ```java + public class Products_ByProductType extends AbstractIndexCreationTask { + public Products_ByProductType() { + + // Call 'CreateField' to generate the dynamic-index-field. + // The field name will be the value of document field 'productType'. + // The field terms will be derived from document field 'pricePerUnit'. + map = "docs.Products.Select(p => new { " + + " _ = this.CreateField(p.productType, p.pricePerUnit) " + + "})"; + } + } + ``` + + + ```java + public class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByProductType_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + // The field name will be the value of document field 'productType' + // The field terms will be derived from document field 'pricePerUnit' + " _: createField(p.productType, p.pricePerUnit) " + + " }; " + + "})" + )); + } + } + ``` + + - -Consider if that is a true necessity, as indexing every single field can end up costing time and disk space. +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Products_ByProductType.class) + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + .whereEquals("Electronics", 23) + .toList(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + + +### Example - list + + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. * **The document**: - - -{`public class Product \{ - private String id; - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - private String firstName; - private String lastName; - private String title; - // ... - - // get + set implementation ... -\} -`} - - - - - -{`// Sample document content -\{ - "firstName": "John", - "lastName": "Doe", - "title": "Engineer", - // ... -\} -`} - - + + + ```java + public class Product { + private String id; + private String name; + + // For each element in this list, + // the VALUE of property 'propName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + private List attributes; + + // getters and setters ... + } -* **The index**: - The following index will index any field from the document, - a dynamic-index-field will be created for each field. - New fields added to the document after index creation time will be dynamically indexed as well. + public class Attribute { + private String propName; + private String propValue; - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "name": "SomeName", + "attributes": [ + { + "propName": "Color", + "propValue": "Blue" + }, + { + "propName": "Width", + "propValue": "10" + }, + { + "propName": "Length", + "propValue": "20" + } + // ... + ] + } + ``` + - - - -{`public class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { - public Products_ByAnyField_JS() { +* **The index**: - // This will index EVERY FIELD under the top level of the document - setMaps(Sets.newHashSet( - "map('Products', function (p) { " + - " return { " + - " _: Object.keys(p).map(key => createField(key, p[key], " + - " { indexing: 'Search', storage: true, termVector: null })) " + - " }; " + - "}) " - )); + The following index will create a dynamic-index-field per item in the document's `attributes` list. + New items added to the attributes list after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the item's propName **value**. + E.g., 'propName' value `Width` will be a dynamic-index-field. + + + + ```java + public class Attributes_ByName extends AbstractIndexCreationTask { + public Attributes_ByName() { + + // Define the dynamic-index-fields by calling 'CreateField'. + // A dynamic-index-field will be generated for each item in 'attributes'. + + // For each item, the field name will be the value of field 'propName' + // The field terms will be derived from field 'propValue'. + // A regular index field (Name) is defined as well. + map = "docs.Products.Select(p => new { " + + " _ = p.attributes.Select(item => " + + " this.CreateField(item.propName, item.propValue)), " + + " Name = p.name " + + "})"; + } } -} -`} - - - + ``` + + + ```java + public class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask { + public Attributes_ByName_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + // For each item, + // the field name will be the value of field 'propName'. + // The field terms will be derived from field 'propValue'. + " _: p.attributes.map(item => " + + " createField(item.propName, item.propValue)), " + + // A regular index field can be defined as well: + " Name: p.name " + + " }; " + + "})" + )); + } + } + ``` + + * **The query**: - * To get all documents with some 'lastName' use: - - - -{`List matchingDocuments = session - .query(Product.class, Products_ByAnyField_JS.class) - .whereEquals("lastName", "Doe") - .toList(); -`} - - - - -{`// 'lastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where lastName = "Doe" -`} - - - + + To get all documents matching a specific attribute property use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Attributes_ByName.class) + // 'Width' is a dynamic-index-field that was indexed from the attributes list + .whereEquals("Width", 10) + .toList(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `indexing: 'Search'` in the `createField` options object (`AbstractJavaScriptIndexCreationTask`) +or `FieldIndexing.Search` in the `CreateFieldOptions` (`AbstractIndexCreationTask`). +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. -## Indexing documents fields VALUES + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). -## Example - basic +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. - + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. -* For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---index-a-list-of-properties) below. + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: -* **The document**: - - -{`public class Product \{ - private String id; - - // The VALUE of productType will be dynamically indexed - private String productType; - private int pricePerUnit; - - // get + set implementation ... -\} -`} - - + + ```java + public class Product { + private String id; - - -{`// Sample document content -\{ - "productType": "Electronics", - "pricePerUnit": 23 -\} -`} - - + private Map descriptions; -* **The index**: - The following index will index the **value** of document field 'productType'. + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `descriptions` map. + For example, the key `English` creates the dynamic field `Description_English`. + + Each generated dynamic field is indexed for **full-text search**. + + + + ```java + public class Products_ByLocalizedDescription extends AbstractIndexCreationTask { + + private static final String USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription() { + map = "docs.Products.Select(product => new { " + + " _ = product.descriptions.Select(x => " + + " this.CreateField( " + + " \"Description_\" + x.Key, " + + " x.Value, " + + " new CreateFieldOptions { " + + // Index each generated dynamic field for FTS + " Indexing = FieldIndexing.Search, " + + " Storage = FieldStorage.No " + + " })) " + + "})"; + + storeAllFields(FieldStorage.NO); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + getConfiguration().put(USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS, "true"); + } + } + ``` + + + ```java + public class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + + private static final String USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (product) { " + + " return { " + + " _: Object.keys(product.descriptions) " + + " .map(function (key) { " + + " return createField( " + + " 'Description_' + key, " + + " product.descriptions[key], " + + " { " + + // Index each generated dynamic field for FTS + " indexing: 'Search', " + + " storage: false " + + " }); " + + " }) " + + " }; " + + "})" + )); + + Map fields = new HashMap<>(); + + IndexFieldOptions allFieldsOptions = new IndexFieldOptions(); + allFieldsOptions.setStorage(FieldStorage.NO); + + fields.put(Constants.Documents.Indexing.Fields.ALL_FIELDS, allFieldsOptions); + setFields(fields); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + getConfiguration().put(USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS, "true"); + } + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```java + List results = session.advanced() + .documentQuery(Product.class, Products_ByLocalizedDescription.class) + .search("Description_English", "north wind") + .toList(); + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + - This value will be the dynamic-index-field name on which you can query. - e.g. Field value `Electronics` will be the dynamic-index-field. +--- - - - -{`public class Products_ByProductType extends AbstractIndexCreationTask { - public Products_ByProductType() { + + +### Configuring analyzers for dynamic fields + +The `CreateFieldOptions` (and the JavaScript `createField` options) do **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: - // The field name will be the value of document field 'productType' - // The field terms will be derived from document field 'pricePerUnit' - map = "docs.Products.Select(p => new { " + - " _ = this.CreateField(p.productType, p.pricePerUnit) " + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field + + + +```java +public class Products_ByLocalizedDescription extends AbstractIndexCreationTask { + public Products_ByLocalizedDescription() { + map = "docs.Products.Select(product => new { " + + " _ = product.descriptions.Select(x => " + + " this.CreateField( " + + " \"Description_\" + x.Key, " + + " x.Value, " + + " new CreateFieldOptions { " + + " Indexing = FieldIndexing.Search, " + + " Storage = FieldStorage.No " + + " })) " + "})"; + + storeAllFields(FieldStorage.NO); + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`List matchingDocuments = session - .query(Product.class, Products_ByProductType.class) - .whereEquals("Electronics", 23) - .toList(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'productType' -from index 'Products/ByProductType' where Electronics = 23 -`} - - - - -## Example - list + +```java +public class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByLocalizedDescription_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (product) { " + + " return { " + + " _: Object.keys(product.descriptions) " + + " .map(function (key) { " + + " return createField( " + + " 'Description_' + key, " + + " product.descriptions[key], " + + " { " + + " indexing: 'Search', " + + " storage: false " + + " }); " + + " }) " + + " }; " + + "})" + )); - + Map fields = new HashMap<>(); -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. + IndexFieldOptions allFieldsOptions = new IndexFieldOptions(); + allFieldsOptions.setStorage(FieldStorage.NO); + fields.put(Constants.Documents.Indexing.Fields.ALL_FIELDS, allFieldsOptions); - + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + IndexFieldOptions englishOptions = new IndexFieldOptions(); + englishOptions.setAnalyzer("StopAnalyzer"); + fields.put("Description_English", englishOptions); -* **The document**: - - -{`public class Product \{ - private String id; - private String name; - - // For each element in this list, the VALUE of property 'propName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - private List attributes; - - // get + set implementation ... -\} - -public class Attribute \{ - private String propName; - private String propValue; - - // get + set implementation ... -\} -`} - + setFields(fields); + } +} +``` + - - -{`// Sample document content -\{ -name": "SomeName", -attributes": [ - \{ - "propName": "Color", - "propValue": "Blue" - \}, - \{ - "propName": "Width", - "propValue": "10" - \}, - \{ - "propName": "Length", - "propValue": "20" - \}, - ... - -\} -`} - - +#### Configure a fallback analyzer for all fields + + + +```java +public class Products_ByLocalizedDescription extends AbstractIndexCreationTask { + public Products_ByLocalizedDescription() { + map = "docs.Products.Select(product => new { " + + " _ = product.descriptions.Select(x => " + + " this.CreateField( " + + " \"Description_\" + x.Key, " + + " x.Value, " + + " new CreateFieldOptions { " + + " Indexing = FieldIndexing.Search, " + + " Storage = FieldStorage.No " + + " })) " + + "})"; -* **The index**: - The following index will create a dynamic-index-field per item in the document's `attributes` list. - New items added to the attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's propName **value**. - E.g., 'propName' value `Width` will be a dynamic-index-field. - - - - -{`public class Attributes_ByName extends AbstractIndexCreationTask { - public Attributes_ByName() { - - // For each attribute item, the field name will be the value of field 'propName' - // The field terms will be derived from field 'propValue' - // A regular-index-field (Name) is defined as well - map = - "docs.Products.Select(p => new { " + - " _ = p.attributes.Select(item => this.CreateField(item.propName, item.propValue)), " + - " Name = p.name " + - "})"; + storeAllFields(FieldStorage.NO); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + analyze(Constants.Documents.Indexing.Fields.ALL_FIELDS, "StopAnalyzer"); } } -`} - +``` - + +```java +public class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByLocalizedDescription_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (product) { " + + " return { " + + " _: Object.keys(product.descriptions) " + + " .map(function (key) { " + + " return createField( " + + " 'Description_' + key, " + + " product.descriptions[key], " + + " { " + + " indexing: 'Search', " + + " storage: false " + + " }); " + + " }) " + + " }; " + + "})" + )); -* **The query**: - * To get all documents matching a specific attribute property use: - - - -{`List matchingDocuments = session - .query(Product.class, Attributes_ByName.class) - .whereEquals("Width", 10) - .toList(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the attributes list -from index 'Attributes/ByName' where Width = 10 -`} - + Map fields = new HashMap<>(); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + IndexFieldOptions allFieldsOptions = new IndexFieldOptions(); + allFieldsOptions.setStorage(FieldStorage.NO); + allFieldsOptions.setAnalyzer("StopAnalyzer"); + fields.put(Constants.Documents.Indexing.Fields.ALL_FIELDS, allFieldsOptions); + + setFields(fields); + } +} +``` + + + + +### Which analyzer is used for the query term? +The following applies when a `search()` query targets a **dynamic field**: -## CreateField syntax +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | -#### Syntax for LINQ-index: + - - -{`object CreateField(string name, object value); + + + + +#### Syntax for `AbstractIndexCreationTask` + + +```csharp +object CreateField(string name, object value); object CreateField(string name, object value, bool stored, bool analyzed); object CreateField(string name, object value, CreateFieldOptions options); -`} - +``` -#### Syntax for JavaScript-index: +#### Syntax for `AbstractJavaScriptIndexCreationTask` - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` -| Parameters | Type | Description | -|----------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **fieldName** | `string` | Name of the dynamic-index-field | -| **fieldValue** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorate.Yes` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | +| Parameters | Type | Description | +|----------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorage.Yes` | +| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | | **options** | `CreateFieldOptions` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| | **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +* The `CreateField` syntax (and the `CreateFieldOptions` shape) above describes what is available inside the server-side `map` string of an `AbstractIndexCreationTask`, + since the Java client sends the `map` source to the server for compilation. + The JavaScript `createField` function is the equivalent for `AbstractJavaScriptIndexCreationTask` map sources. + * All above examples have used the character `_` in the dynamic-index-field definition. However, using `_` is just a convention. Any other string can be used instead. * This property is Not queryable, it is only used in the index definition syntax. - The actual dynamic-index-fields that are generated are defined by the `CreateField` method. + The actual dynamic-index-fields that are generated are defined by the `CreateField` method (or the `createField` JS function). + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. - +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-nodejs.mdx b/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-nodejs.mdx index b5df2df4a7..c14184c2b5 100644 --- a/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-nodejs.mdx +++ b/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-nodejs.mdx @@ -2,533 +2,710 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. - -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. - -* In this page: - - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`createField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -## Indexing documents fields KEYS + - -#### Example - index any field under object -The following allows you to: - -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. -**The document**: - - -{`class Product \{ - constructor(id, attributes) \{ - this.id = id; - - // The KEYS under the attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - this.attributes = attributes; - \} -\} -`} - - + + +### Example - index every field under an object - - -{`// Sample document content -\{ - "attributes": \{ - "color": "Red", - "size": 42 - \} -\} -`} - - + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. + -**The index**: +* **The document**: + + + ```js + class Product { + constructor(id, attributes) { + this.id = id; + + // The KEYS under the attributes object will be dynamically indexed + // Fields added to this object after index creation time will also get indexed + this.attributes = attributes; + } + } + ``` + + + + ```json + // Sample document content + { + "attributes": { + "color": "Red", + "size": 42 + } + } + ``` + -* The following index will index any field under the `attributes` object from the document, +* **The index**: + + The following index will index any field under the `attributes` object from the document, a dynamic-index-field will be created for each such field. - New fields added to the object after index creation time will be dynamically indexed as well. - -* The actual dynamic-index-field name on which you can query will be the attribute field **key**. - e.g. Keys `color` & `size` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { - constructor() { - super(); - - const { createField } = this.mapUtils(); - - this.map("Products", p => { - return { - // Call 'createField' to generate dynamic-index-fields from the attributes object keys - // Using '_' is just a convention. Any other string can be used instead of '_' - - // The actual field name will be the key - // The actual field terms will be derived from p.attributes[key] - _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], { - indexing: "Search", - storage: false, - termVector: null - })) - }; - }); + New fields added to the object after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `color` & `size` will become the actual dynamic-index-fields. + + + ```js + class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + this.map("Products", p => { + return { + // Call 'createField' to generate dynamic-index-fields + // from the attributes object keys. + + // Using '_' is just a convention. + // Any other string can be used instead of '_' + + // The actual field name will be the key + // The actual field terms will be derived from p.attributes[key] + _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key])) + }; + }); + } } -} -`} - - - - -**The query**: - -* You can now query the generated dynamic-index fields. - Property `_` is Not queryable, it is only used in the index definition syntax. - -* To get all documents with some 'size' use: - - - - -{`const matchingDocuments = session.query({indexName: 'Products_ByAttributeKey'}) - // 'size' is a dynamic-index-field that was indexed from the attributes object - .whereEquals('size', 42) - .all(); -`} - - - - -{`// 'size' is a dynamic-index-field that was indexed from the attributes object -from index 'Products/ByAttributeKey' where size = 42 -`} - - - - - + ``` + + +* **The query**: + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _size_ use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Products/ByAttributeKey/JS" }) + // 'size' is a dynamic-index-field that was indexed from the attributes object + .whereEquals("size", 42) + .all(); + ``` + + + ```sql + // 'size' is a dynamic-index-field that was indexed from the attributes object + from index 'Products/ByAttributeKey/JS' where size = 42 + ``` + + + + + + + +### Example - index every field -#### Example - index any field -The following allows you to: - -* Define an index on a collection **without** needing any common structure between the indexed documents. -* After index is deployed, any new field added to the document will be indexed as well. - +The following example allows you to: + * Define an index on a collection **without** needing any common structure between the indexed documents. + * After the index is deployed, any new field added to any document in the collection will be indexed as well. + + - -Consider if that is a true necessity, as indexing every single field can end up costing time and disk space. - +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. -**The document**: - - -{`class Product \{ - constructor(id, firstName, lastName, title) \{ - this.id = id; - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - this.firstName = firstName; - this.lastName = lastName; - this.title = title; - // ... - \} -\} -`} - - - - - -{`// Sample document content -\{ - "firstName": "John", - "lastName": "Doe", - "title": "Engineer", - // ... -\} -`} - - -**The index**: +* **The document**: + + + ```js + class Product { + constructor(id, firstName, lastName, title) { + this.id = id; + + // All KEYS in the document will be dynamically indexed + // Fields added to the document after index creation time will also get indexed + this.firstName = firstName; + this.lastName = lastName; + this.title = title; + // ... + } + } + ``` + + + + ```json + // Sample document content + { + "firstName": "John", + "lastName": "Doe", + "title": "Engineer", + // ... + } + ``` + -* The following index will index any field from the document, +* **The index**: + + The following index will index any field from the document, a dynamic-index-field will be created for each field. - New fields added to the document after index creation time will be dynamically indexed as well. - -* The actual dynamic-index-field name on which you can query will be the field **key**. - e.g. Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + New fields added to the document after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + + + ```js + class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + // This will index EVERY FIELD under the top level of the document + this.map("Products", p => { + return { + _: Object.keys(p).map(key => createField(key, p[key])) + }; + }); + } + } + ``` + + +* **The query**: + + To get all documents with a specific _lastName_ use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Products/ByAnyField/JS" }) + // 'lastName' is a dynamic-index-field that was indexed from the document + .whereEquals("lastName", "Doe") + .all(); + ``` + + + ```sql + // 'lastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where lastName = "Doe" + ``` + + + + + + + + + + + +### Example - basic + + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + - - - -{`class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { - constructor () { - super(); +* **The document**: + + + ```js + class Product { + constructor(id, productType, pricePerUnit) { + this.id = id; + + // The VALUE of productType will be dynamically indexed + this.productType = productType; + this.pricePerUnit = pricePerUnit; + } + } + ``` + + + + ```json + // Sample document content + { + "productType": "Electronics", + "pricePerUnit": 23 + } + ``` + + +* **The index**: + + The following index will index the **value** of document field 'productType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + ```js + class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + this.map("Products", p => { + return { + // The field name will be the value of document field 'productType' + // The field terms will be derived from document field 'pricePerUnit' + _: createField(p.productType, p.pricePerUnit) + }; + }); + } + } + ``` + + +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Products/ByProductType/JS" }) + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + .whereEquals("Electronics", 23) + .all(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + from index 'Products/ByProductType/JS' where Electronics = 23 + ``` + + + + + + + +### Example - list - const { createField } = this.mapUtils(); + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. + - this.map("Products", p => { - return { - // This will index EVERY FIELD under the top level of the document - _: Object.keys(p).map(key => createField(key, p[key], { - indexing: "Search", - storage: true, - termVector: null - })) - }; - }); +* **The document**: + + + ```js + class Product { + constructor(id, name, attributes) { + this.id = id; + this.name = name; + + // For each element in this list, + // the VALUE of property 'propName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + this.attributes = attributes; + } } -} -`} - - - -**The query**: + class Attribute { + constructor(propName, propValue) { + this.propName = propName; + this.propValue = propValue; + } + } + ``` + + + + ```json + // Sample document content + { + "name": "SomeName", + "attributes": [ + { + "propName": "Color", + "propValue": "Blue" + }, + { + "propName": "Width", + "propValue": "10" + }, + { + "propName": "Length", + "propValue": "20" + }, + ... + ] + } + ``` + + +* **The index**: + + The following index will create a dynamic-index-field per item in the document's `attributes` list. + New items added to the attributes list after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the item's propName **value**. + E.g., 'propName' value `Width` will be a dynamic-index-field. + + + ```js + class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + this.map("Products", p => { + return { + // Define the dynamic-index-fields by calling 'createField' + // A dynamic-index-field will be generated for each item in 'attributes' + + // For each item, the field name will be the value of field 'propName' + // The field terms will be derived from field 'propValue' + _: p.attributes.map(item => + createField(item.propName, item.propValue)), + + // A regular index field can be defined as well: + name: p.name + }; + }); + } + } + ``` + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Attributes/ByName/JS" }) + // 'Width' is a dynamic-index-field that was indexed from the attributes list + .whereEquals("Width", 10) + .all(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the attributes list + from index 'Attributes/ByName/JS' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `indexing: "Search"` in the `createField` options. + +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. + + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. + + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. -* To get all documents with some 'lastName' use: + + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```js + class Product { + constructor(id, descriptions) { + this.id = id; + this.descriptions = descriptions; + } + } + ``` + + + + ```json + // Sample document content + { + "descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. + + Each generated dynamic field is indexed for **full-text search**. + + + ```js + class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + const useSearchAnalyzerForDynamicFields = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + this.map("Products", product => { + return { + _: Object.keys(product.descriptions).map(key => + createField( + "Description_" + key, + product.descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: "Search", + storage: false + })) + }; + }); + + this.storeAllFields("No"); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + this.configuration[useSearchAnalyzerForDynamicFields] = "true"; + } + } + ``` + - - - -{`const matchingDocuments = session.query({ indexName: 'Products_ByAnyField_JS' }) - // 'lastName' is a dynamic-index-field that was indexed from the document - .whereEquals('lastName', 'Doe') - .all(); -`} - - - - -{`// 'lastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where lastName = "Doe" -`} - - - +* **Full-text search query**: - + Query the generated dynamic field by its field name. + In this example, the query targets the generated field `Description_English`. + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). -## Indexing documents fields VALUES + + + ```js + const results = await session + .query({ indexName: "Products/ByLocalizedDescription/JS" }) + .search("Description_English", "north wind") + .all(); + ``` + - -#### Example - basic -This example shows: - -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. -* For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---index-a-list-of-properties) below. -**The document**: - - -{`class Product \{ - constructor(id, productType, pricePerUnit) \{ - this.id = id; - - // The VALUE of productType will be dynamically indexed - this.productType = productType; - this.pricePerUnit = pricePerUnit; - \} -\} -`} - - + + ```sql + from index "Products/ByLocalizedDescription/JS" + where search(Description_English, "north wind") + ``` + + - - -{`// Sample document content -\{ - "productType": "Electronics", - "pricePerUnit": 23 -\} -`} - - +--- -**The index**: + + +### Configuring analyzers for dynamic fields -* The following index will index the **value** of document field 'productType'. +The `createField` options object does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. -* This value will be the dynamic-index-field name on which you can query. - e.g. Field value `Electronics` will be the dynamic-index-field. +--- - - - -{`class Products_ByProductType extends AbstractCsharpIndexCreationTask { - constructor () { - super(); +#### Configure analyzer for a specific dynamic field - // The field name will be the value of document field 'productType' - // The field terms will be derived from document field 'pricePerUnit' - this.map = "docs.Products.Select(p => new { " + - " _ = this.CreateField(p.productType, p.pricePerUnit) " + - "})"; - } -} -`} - - - - -{`class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask { - constructor () { + +```js +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { super(); const { createField } = this.mapUtils(); - this.map("Products", p => { + this.map("Products", product => { return { - _: [ - // The field name will be the value of document field 'productType' - // The field terms will be derived from document field 'pricePerUnit' - createField(p.productType, p.pricePerUnit, { - indexing: "Search", - storage: false, - termVector: null - }) - ] + _: Object.keys(product.descriptions).map(key => + createField( + "Description_" + key, + product.descriptions[key], + { + indexing: "Search", + storage: false + })) }; }); - } -} -`} - - - - -**The query**: - -* To get all documents of some product type having a specific price per unit use: - - - - -{`const matchingDocuments = session.query({ indexName: 'Products_ByProductType' }) - // 'Electronics' is the dynamic-index-field that was indexed from document field 'productType' - .whereEquals('Electronics', 23) - .all(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'productType' -from index 'Products/ByProductType' where Electronics = 23 -`} - - - - - - -#### Example - list -The following allows you to: - -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. -**The document**: - - -{`class Product \{ - constructor(id, name, attributes) \{ - this.id = id; - this.name = name; - - // For each element in this list, the VALUE of property 'propName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - this.attributes = attributes; - \} -\} - -class Attribute \{ - constructor(propName, propValue) \{ - this.propName = propName; - this.propValue = propValue; - \} -\} -`} - - - - - -{`// Sample document content -\{ - "name": "SomeName", - "attributes": [ - \{ - "propName": "Color", - "propValue": "Blue" - \}, - \{ - "propName": "Width", - "propValue": "10" - \}, - \{ - "propName": "Length", - "propValue": "20" - \}, - ... - ] -\} -`} - - - -**The index**: - -* The following index will create a dynamic-index-field per item in the document's `attributes` list. - New items added to the attributes list after index creation time will be dynamically indexed as well. - -* The actual dynamic-index-field name on which you can query will be the item's propName **value**. - e.g. 'propName' value `Width` will be a dynamic-index-field. - - - - -{`class Attributes_ByName extends AbstractCsharpIndexCreationTask -{ - constructor () { - super(); + this.storeAllFields("No"); - // For each attribute item, the field name will be the value of field 'propName' - // The field terms will be derived from field 'propValue' - // A regular-index-field (Name) is defined as well - this.map = - "docs.Products.Select(p => new { " + - " _ = p.attributes.Select(item => this.CreateField(item.propName, item.propValue)), " + - " Name = p.name " + - "})"; + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + this.analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -{`class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask { - constructor () { + +#### Configure a fallback analyzer for all fields + + +```js +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { super(); const { createField } = this.mapUtils(); - this.map("Products", p => { + this.map("Products", product => { return { - // For each item, the field name will be the value of field 'propName' - // The field terms will be derived from field 'propValue' - _: p.attributes.map(item => createField(item.propName, item.propValue, { - indexing: "Search", - storage: true, - termVector: null - })), - - // A regular-index-field can be defined as well: - Name: p.name + _: Object.keys(product.descriptions).map(key => + createField( + "Description_" + key, + product.descriptions[key], + { + indexing: "Search", + storage: false + })) }; }); - } -} -`} - - - - -**The query**: -* To get all documents matching a specific attribute property use: + this.storeAllFields("No"); - - - -{`const matchingDocuments = session.query({ indexName: 'Attributes/ByName' }) - // 'Width' is a dynamic-index-field that was indexed from the attributes list - .whereEquals('Width', 10) - .all(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the attributes list -from index 'Attributes/ByName' where Width = 10 -`} - + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + this.analyze(CONSTANTS.Documents.Indexing.Fields.ALL_FIELDS, "StopAnalyzer"); + } +} +``` - - - + + + + +### Which analyzer is used for the query term? -## CreateField syntax +The following applies when a `search()` query targets a **dynamic field**: -#### Syntax for LINQ-index: +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `this.analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `this.analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | - - -{`object CreateField(string name, object value); + -object CreateField(string name, object value, bool stored, bool analyzed); + -object CreateField(string name, object value, CreateFieldOptions options); -`} - - + -#### Syntax for JavaScript-index: +#### Syntax: - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` -| Parameters | Type | Description | -|----------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **fieldName** | `string` | Name of the dynamic-index-field | -| **fieldValue** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorate.Yes` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | -| **options** | `CreateFieldOptions` | Dynamic-index-field options | +| Parameters | Type | Description | +|----------------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **options** | `object` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| -| **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| options object | | | +|-----------------|--------------------|-----------------------------------------------------------------------------------| +| **storage** | `boolean` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | +| **indexing** | `FieldIndexing` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **termVector** | `FieldTermVector` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -536,21 +713,19 @@ object CreateField(string name, object value, CreateFieldOptions options); However, using `_` is just a convention. Any other string can be used instead. * This property is Not queryable, it is only used in the index definition syntax. - The actual dynamic-index-fields that are generated are defined by the `CreateField` method. + The actual dynamic-index-fields that are generated are defined by the `createField` method. + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. - +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-php.mdx b/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-php.mdx index 09c58cc9fc..6992310715 100644 --- a/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-php.mdx +++ b/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-php.mdx @@ -2,537 +2,969 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. + -* In this page: + - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) + + +### Example - index every field under an object + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. -## Indexing documents fields KEYS - -## Example - index any field under object +* **The document**: + + + ```php + use Ds\Map as DSMap; - + class Product + { + public ?string $id = null; -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + // The KEYS under the attributes object will be dynamically indexed. + // Fields added to this object after index creation time will also get indexed. + public ?DSMap $attributes = null; - - -* **The document**: - - -{`use Ds\\Map as DSMap; - -class Product -\{ - private ?string $id = null; - - // The KEYS under the Attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - public ?DSMap $attributes = null; -\} -`} - - - - -{`// Sample document content -\{ - "Attributes": \{ - "Color": "Red", - "Size": 42 - \} -\} -`} - - + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "attributes": { + "Color": "Red", + "Size": 42 + } + } + ``` + * **The index**: - The below index will index any field under the `Attributes` object from the document, + + The following index will index any field under the `attributes` object from the document, a dynamic-index-field will be created for each such field. New fields added to the object after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. - - - -{`class Products_ByAttributeKey extends AbstractIndexCreationTask -{ - public function __construct() + + + ```php + class Products_ByAttributeKey extends AbstractIndexCreationTask { - parent::__construct(); - - $this->map = "from p in docs.Products select new {" . - "_ = p.attributes.Select(item => CreateField(item.Key, item.Value))" . - "}"; + public function __construct() + { + parent::__construct(); + + // Call 'CreateField' to generate dynamic-index-fields + // from the attributes object keys. + // + // Using '_' is just a convention. + // Any other string can be used instead of '_'. + // + // The actual field name will be 'item.Key'. + // The actual field terms will be derived from 'item.Value'. + $this->map = + "from p in docs.Products " . + "select new { " . + " _ = p.attributes.Select(item => CreateField(item.Key, item.Value)) " . + "}"; + } } -} -`} - - - - -{`class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask -{ - public function __construct() + ``` + + + ```php + class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { - parent::__construct(); - - $this->setMaps([ - "map('Products', function (p) { " . - " return { " . - " _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], " . - " { indexing: 'Search', storage: false, termVector: null })) " . - " }; " . - "}) " - ]); + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (p) { + return { + // The field name will be the key + // The field terms will be derived from the corresponding value + _: Object.keys(p.attributes) + .map(key => createField(key, p.attributes[key], null)) + }; + })" + ]); + } } -} -`} - - - - + ``` + + * **The query**: - * You can now query the generated dynamic-index fields. - * To get all documents with some 'size' use: - - - -{`$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Products_ByAttributeKey::class) - // 'Size' is a dynamic-index-field that was indexed from the Attributes object - ->whereEquals("Size", 42) - ->toList(); -`} - - - - -{`// 'Size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey' where Size = 42 -`} - - - - -## Example - index any field - - + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _Size_ use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Products_ByAttributeKey::class) + // 'Size' is a dynamic-index-field that was indexed from the attributes object + ->whereEquals("Size", 42) + ->toList(); + ``` + + + ```sql + // 'Size' is a dynamic-index-field that was indexed from the attributes object + from index 'Products/ByAttributeKey' where Size = 42 + ``` + + + + + Dynamic index fields are queried with _documentQuery_ or _RQL_ because the field names (e.g. `Size`, `Color`) + are generated at indexing time and are not properties on the _Product_ class. + + Use the string-based `whereEquals("Size", 42)` or RQL `where Size = 42` instead. + + + + + + +### Example - index every field + +The following example allows you to: * Define an index on a collection **without** needing any common structure between the indexed documents. - * After index is deployed, any new field added to the document will be indexed as well. - + * After the index is deployed, any new field added to any document in the collection will be indexed as well. - + -Consider whether this is really necessary, as indexing every single field can end up costing time and disk space. +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`class Product -\{ - private ?string $id = null; - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - public ?string $firstName = null; - public ?string $lastName = null; - public ?string $title = null; - - // ... getters and setters -\} -`} - - + + + ```php + class Product + { + public ?string $id = null; - - -{`// Sample document content - \{ - "FirstName": "John", - "LastName": "Doe", - "Title": "Engineer", + // All KEYS in the document will be dynamically indexed. + // Fields added to the document after index creation time will also get indexed. + public ?string $firstName = null; + public ?string $lastName = null; + public ?string $title = null; // ... -\} -`} - - + + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "firstName": "John", + "lastName": "Doe", + "title": "Engineer" + // ... + } + ``` + * **The index**: - The below index will index any field from the document, + + The following index will index any field from the document, a dynamic-index-field will be created for each field. New fields added to the document after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask -{ - public function __construct() + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + + The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection. + + + + ```php + class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { - parent::__construct(); - - // This will index EVERY FIELD under the top level of the document - $this->setMaps([ - "map('Products', function (p) { - return { - _: Object.keys(p).map(key => createField(key, p[key], - { indexing: 'Search', storage: true, termVector: null })) - } - })" - ]); + public function __construct() + { + parent::__construct(); + + // This will index EVERY FIELD under the top level of the document + $this->setMaps([ + "map('Products', function (p) { + return { + _: Object.keys(p).map(key => createField(key, p[key], null)) + }; + })" + ]); + } } -} -`} - - - + ``` + + * **The query**: - * To get all documents with some 'LastName' use: - - - -{`$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Products_ByAnyField_JS::class) - // 'LastName' is a dynamic-index-field that was indexed from the document - ->whereEquals("LastName", "Doe") - ->toList(); -`} - - - - -{`// 'LastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where LastName = "Doe" -`} - - - + + To get all documents with a specific _lastName_ use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Products_ByAnyField_JS::class) + // 'lastName' is a dynamic-index-field that was indexed from the document + ->whereEquals("lastName", "Doe") + ->toList(); + ``` + + + ```sql + // 'lastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where lastName = "Doe" + ``` + + + + + + + + + + + +### Example - basic + + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + +* **The document**: + + + ```php + class Product + { + public ?string $id = null; + // The VALUE of productType will be dynamically indexed + public ?string $productType = null; + public ?int $pricePerUnit = null; -## Indexing documents fields VALUES + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "productType": "Electronics", + "pricePerUnit": 23 + } + ``` + -## Example - basic +* **The index**: + + The following index will index the **value** of document field 'productType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. - + + + ```php + class Products_ByProductType extends AbstractIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + // Call 'CreateField' to generate the dynamic-index-fields. + // The field name will be the value of document field 'productType'. + // The field terms will be derived from document field 'pricePerUnit'. + $this->map = + "from p in docs.Products " . + "select new { " . + " _ = CreateField(p.productType, p.pricePerUnit) " . + "}"; + } + } + ``` + + + ```php + class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (p) { + return { + // The field name will be the value of document field 'productType' + // The field terms will be derived from document field 'pricePerUnit' + _: createField(p.productType, p.pricePerUnit, null) + }; + })" + ]); + } + } + ``` + + -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Products_ByProductType::class) + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + ->whereEquals("Electronics", 23) + ->toList(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + + +### Example - list + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. * **The document**: - - -{`class Product -\{ - public ?string $id = null; - - // The VALUE of ProductType will be dynamically indexed - public ?string $productType = null; - public ?int $pricePerUnit = null; - - // ... getters and setters -\} -`} - - + + + ```php + class Product + { + public ?string $id = null; + public ?string $name = null; - - -{`// Sample document content -\{ - "ProductType": "Electronics", - "PricePerUnit": 23 -\} -`} - - + // For each element in this list, + // the VALUE of property 'propName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + public ?AttributeList $attributes = null; + + // ... getters and setters + } + + class Attribute + { + public ?string $propName = null; + public ?string $propValue = null; + + // ... getters and setters + } + + class AttributeList extends TypedList + { + protected function __construct() + { + parent::__construct(Attribute::class); + } + } + ``` + + + + ```json + // Sample document content + { + "name": "SomeName", + "attributes": [ + { + "propName": "Color", + "propValue": "Blue" + }, + { + "propName": "Width", + "propValue": "10" + }, + { + "propName": "Length", + "propValue": "20" + } + // ... + ] + } + ``` + * **The index**: - The below index will index the **value** of document field 'ProductType'. + + The following index will create a dynamic-index-field per item in the document's `attributes` list. + New items added to the attributes list after index creation time will be dynamically indexed as well. - This value will be the dynamic-index-field name on which you can query. - E.g., Field value `Electronics` will be the dynamic-index-field. + The actual dynamic-index-field name on which you can query will be the item's propName **value**. + E.g., 'propName' value `Width` will be a dynamic-index-field. + + + + ```php + class Attributes_ByName extends AbstractIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + // Define the dynamic-index-fields by calling 'CreateField'. + // A dynamic-index-field will be generated for each item in 'attributes'. + // + // For each item, the field name will be the value of field 'propName'. + // The field terms will be derived from field 'propValue'. + $this->map = + "from p in docs.Products " . + "select new { " . + " _ = p.attributes.Select(item => CreateField(item.propName, item.propValue)), " . + " Name = p.name " . + "}"; + } + } + ``` + + + ```php + class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (p) { + return { + // For each item, + // the field name will be the value of field 'propName' + // The field terms will be derived from field 'propValue' + _: p.attributes.map(item => createField( + item.propName, item.propValue, null)), + + // A regular index field can be defined as well: + Name: p.name + }; + })" + ]); + } + } + ``` + + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Attributes_ByName::class) + // 'Width' is a dynamic-index-field that was indexed from the attributes list + ->whereEquals("Width", 10) + ->toList(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `FieldIndexing::search()` in `CreateFieldOptions`. + +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. + + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. + + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + + + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```php + use Ds\Map as DSMap; + + class Product + { + public ?string $id = null; + public ?DSMap $descriptions = null; + + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. - + Each generated dynamic field is indexed for **full-text search**. + + + + ```php + class Products_ByLocalizedDescription extends AbstractIndexCreationTask + { + private const USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public function __construct() + { + parent::__construct(); + + $this->map = + "from product in docs.Products " . + "select new { " . + " _ = product.descriptions.Select(x => " . + " CreateField(" . + " \"Description_\" + x.Key, " . + " x.Value, " . + " new CreateFieldOptions { " . + // Index each generated dynamic field for FTS + " Indexing = FieldIndexing.Search, " . + " Storage = FieldStorage.No " . + " })) " . + "}"; + + $this->storeAllFields(FieldStorage::no()); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + $this->getConfiguration()[self::USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS] = "true"; + } + } + ``` + + + ```php + class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask + { + private const USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (product) { + return { + _: Object.keys(product.descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: 'Search', + storage: 'No' + }); + }) + }; + })" + ]); + + // Apply a storage default to every generated field + $fields = new IndexFieldOptionsArray(); + $allFieldsOptions = new IndexFieldOptions(); + $allFieldsOptions->setStorage(FieldStorage::no()); + $fields->offsetSet(DocumentsIndexingFields::ALL_FIELDS, $allFieldsOptions); + $this->setFields($fields); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + $this->getConfiguration()[self::USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS] = "true"; + } + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```php + /** @var array $results */ + $results = $session->advanced() + ->documentQuery(Product::class, Products_ByLocalizedDescription::class) + ->search("Description_English", "north wind") + ->toList(); + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + + +--- + + + +### Configuring analyzers for dynamic fields + +`CreateFieldOptions` does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field + + - -{`class Products_ByProductType extends AbstractIndexCreationTask +```php +class Products_ByLocalizedDescription extends AbstractIndexCreationTask { public function __construct() { parent::__construct(); - // Call 'CreateField' to generate the dynamic-index-fields - // The field name will be the value of document field 'ProductType' - // The field terms will be derived from document field 'PricePerUnit' - $this->map = "docs.Products.Select(p => new { " . - " _ = this.CreateField(p.productType, p.pricePerUnit) " . - "})"; + $this->map = + "from product in docs.Products " . + "select new { " . + " _ = product.descriptions.Select(x => " . + " CreateField(" . + " \"Description_\" + x.Key, " . + " x.Value, " . + " new CreateFieldOptions { " . + " Indexing = FieldIndexing.Search, " . + " Storage = FieldStorage.No " . + " })) " . + "}"; + + $this->storeAllFields(FieldStorage::no()); + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + $this->analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -{`class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask + +```php +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { public function __construct() { parent::__construct(); $this->setMaps([ - "map('Products', function (p) { + "map('Products', function (product) { return { - _: createField(p.ProductType, p.PricePerUnit, - { indexing: 'Search', storage: true, termVector: null }) + _: Object.keys(product.descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" ]); - } -} -`} - - - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Products_ByProductType::class) -// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -->whereEquals("Electronics", 23) -->toList(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -from index 'Products/ByProductType' where Electronics = 23 -`} - - - - -## Example - list - - -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. + // Apply a storage default to every generated field + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + $fields = new IndexFieldOptionsArray(); - + $allFieldsOptions = new IndexFieldOptions(); + $allFieldsOptions->setStorage(FieldStorage::no()); + $fields->offsetSet(DocumentsIndexingFields::ALL_FIELDS, $allFieldsOptions); -* **The document**: - - -{`class Product -\{ - public ?string $id = null; - public ?string $name = null; - - // For each element in this list, the VALUE of property 'PropName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - public ?AttributeList $attributes = null; - - // ... getters and setters -\} - -class Attribute -\{ - public ?string $propName = null; - public ?string $propValue = null; - - // ... getters and setters -\} - -class AttributeList extends TypedList -\{ - protected function __construct() - \{ - parent::__construct(Attribute::class); - \} -\} -`} - - + $englishFieldOptions = new IndexFieldOptions(); + $englishFieldOptions->setAnalyzer("StopAnalyzer"); + $fields->offsetSet("Description_English", $englishFieldOptions); - - -{`// Sample document content -\{ -Name": "SomeName", -Attributes": [ - \{ - "PropName": "Color", - "PropValue": "Blue" - \}, - \{ - "PropName": "Width", - "PropValue": "10" - \}, - \{ - "PropName": "Length", - "PropValue": "20" - \}, - ... - -\} -`} - + $this->setFields($fields); + } +} +``` + -* **The index**: - The below index will create a dynamic-index-field per item in the document's `Attributes` list. - New items added to the Attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's PropName **value**. - E.g., 'PropName' value `width` will be a dynamic-index-field. +#### Configure a fallback analyzer for all fields - + - -{`class Attributes_ByName extends AbstractIndexCreationTask +```php +class Products_ByLocalizedDescription extends AbstractIndexCreationTask { public function __construct() { parent::__construct(); - // Define the dynamic-index-fields by calling CreateField - // A dynamic-index-field will be generated for each item in the Attributes list + $this->map = + "from product in docs.Products " . + "select new { " . + " _ = product.descriptions.Select(x => " . + " CreateField(" . + " \"Description_\" + x.Key, " . + " x.Value, " . + " new CreateFieldOptions { " . + " Indexing = FieldIndexing.Search, " . + " Storage = FieldStorage.No " . + " })) " . + "}"; - // For each item, the field name will be the value of field 'PropName' - // The field terms will be derived from field 'PropValue' + $this->storeAllFields(FieldStorage::no()); - $this->map = - "docs.Products.Select(p => new { " . - " _ = p.attributes.Select(item => this.CreateField(item.propName, item.propValue)), " . - " Name = p.name " . - "})"; + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + $this->analyze(DocumentsIndexingFields::ALL_FIELDS, "StopAnalyzer"); } } -`} - +``` - - -{`class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask + +```php +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { public function __construct() { parent::__construct(); $this->setMaps([ - "map('Products', function (p) { + "map('Products', function (product) { return { - _: p.Attributes.map(item => createField(item.PropName, item.PropValue, - { indexing: 'Search', storage: true, termVector: null })), - Name: p.Name + _: Object.keys(product.descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" ]); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + $fields = new IndexFieldOptionsArray(); + $allFieldsOptions = new IndexFieldOptions(); + $allFieldsOptions->setStorage(FieldStorage::no()); + $allFieldsOptions->setAnalyzer("StopAnalyzer"); + $fields->offsetSet(DocumentsIndexingFields::ALL_FIELDS, $allFieldsOptions); + $this->setFields($fields); } } -`} - - - - -* **The query**: - To get all documents matching a specific attribute property use: - - - -{`/** @var array $matchingDocuments */ -$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Attributes_ByName::class) - // 'Width' is a dynamic-index-field that was indexed from the Attributes list - ->whereEquals("Width", 10) - ->toList(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the Attributes list -from index 'Attributes/ByName' where Width = 10 -`} - +``` + + + + +### Which analyzer is used for the query term? +The following applies when a `search()` query targets a **dynamic field**: -## CreateField syntax +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `$this->analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `$this->analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | - - -{`object CreateField(string name, object value); + + + + + + +#### Syntax for LINQ-index: + + +```php +object CreateField(string name, object value); object CreateField(string name, object value, bool stored, bool analyzed); object CreateField(string name, object value, CreateFieldOptions options); -`} - +``` -| Parameters | Type | Description | -|------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **name** | `string` | Name of the dynamic-index-field | -| **value** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`False` - will set `FieldStorage.NO` (default value)
`True` - will set `FieldStorate.YES` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`None` - `FieldIndexing.Default` (default value)
`False` - `FieldIndexing.Exact`
`True` - `FieldIndexing.Search` | -| **options** | `CreateFieldOptions` | Dynamic-index-field options | +#### Syntax for JavaScript-index: -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| -| **Storage** | `FieldStorage` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing` | | -| **TermVector** | `FieldTermVector` | | + +```js +createField(fieldName, fieldValue, options); // returns object +``` + + +| Parameters | Type | Description | +|----------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage::no()` (default value)
`true` - will set `FieldStorage::yes()` | +| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing::default()` (default value)
`false` - `FieldIndexing::exact()`
`true` - `FieldIndexing::search()` | +| **options** | `CreateFieldOptions` | Dynamic-index-field options | + +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| +| **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -544,17 +976,15 @@ object CreateField(string name, object value, CreateFieldOptions options); + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. - +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-python.mdx b/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-python.mdx index 69514a29e2..6862d18067 100644 --- a/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-python.mdx +++ b/versioned_docs/version-7.0/indexes/content/_using-dynamic-fields-python.mdx @@ -1,490 +1,886 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. - -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. - -* In this page: - - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -## Indexing documents fields KEYS - -## Example - index any field under object + - - -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + + +### Example - index every field under an object + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. * **The document**: - - -{`class Product: - def __init__(self, Id: str = None, attributes: Dict[str, object] = None): - self.Id = Id - - # The KEYS under the Attributes object will be dynamically indexed - # Fields added to this object after index creation time will also get indexed - self.attributes = attributes -`} - - - - -{`// Sample document content -\{ - "Attributes": \{ - "Color": "Red", - "Size": 42 - \} -\} -`} - - + + + ```python + class Product: + def __init__(self, Id: str = None, Attributes: Dict[str, object] = None): + self.Id = Id + + # The KEYS under the Attributes object will be dynamically indexed + # Fields added to this object after index creation time will also get indexed + self.Attributes = Attributes + ``` + + + + ```json + // Sample document content + { + "Attributes": { + "Color": "Red", + "Size": 42 + } + } + ``` + * **The index**: - The below index will index any field under the `Attributes` object from the document, + + The following index will index any field under the `Attributes` object from the document, a dynamic-index-field will be created for each such field. New fields added to the object after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAttributeKey(AbstractIndexCreationTask): - def __init__(self): - super().__init__() - self.map = ( - "from p in docs.Products select new {" - "_ = p.attributes.Select(item => CreateField(item.Key, item.Value))" - "}" - ) -`} - - - - -{`class Products_ByAttributeKey_JS(AbstractJavaScriptIndexCreationTask): - def __init__(self): - super().__init__() - self.maps = { - """ - map('Products', function (p) { - return { - _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], - { indexing: 'Search', storage: true, termVector: null })) - }; - }) - """ - } -`} - - - - + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. + + + + ```python + class Products_ByAttributeKey(AbstractIndexCreationTask): + def __init__(self): + super().__init__() + self.map = ( + "from p in docs.Products select new { " + # Call 'CreateField' to generate dynamic-index-fields + # from the Attributes object keys. + + # Using '_' is just a convention. + # Any other string can be used instead of '_' + + # The actual field name will be 'item.Key' + # The actual field terms will be derived from 'item.Value' + "_ = p.Attributes.Select(item => CreateField(item.Key, item.Value)) " + "}" + ) + ``` + + + ```python + class Products_ByAttributeKey_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (p) { + return { + // The field name will be the key + // The field terms will be derived from the corresponding value + _: Object.keys(p.Attributes) + .map(key => createField(key, p.Attributes[key])) + }; + }) + """ + ] + ``` + + * **The query**: - * You can now query the generated dynamic-index fields. - * To get all documents with some 'size' use: - - - -{`matching_documents = list( - session.query_index_type(Products_ByAttributeKey, Product) - # 'size' is a dynamic-index-field that was indexed from the attributes object - .where_equals("size", 42) -) -`} - - - - -{`// 'Size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey' where Size = 42 -`} - - - - -## Example - index any field - - + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _Size_ use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Products_ByAttributeKey, Product) + # 'Size' is a dynamic-index-field that was indexed from the Attributes object + .where_equals("Size", 42) + ) + ``` + + + ```sql + // 'Size' is a dynamic-index-field that was indexed from the Attributes object + from index 'Products/ByAttributeKey' where Size = 42 + ``` + + + + + Dynamic index fields are queried with _document_query_ or _RQL_ because the field names (e.g. `Size`, `Color`) + are generated at indexing time and are not properties on the _Product_ class. + + Use the string-based `where_equals("Size", 42)` or RQL `where Size = 42`. + + + + + + +### Example - index every field + +The following example allows you to: * Define an index on a collection **without** needing any common structure between the indexed documents. - * After index is deployed, any new field added to the document will be indexed as well. - + * After the index is deployed, any new field added to any document in the collection will be indexed as well. - + -Consider whether this is really necessary, as indexing every single field can end up costing time and disk space. +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`class Product: - def __init__(self, Id: str = None, first_name: str = None, last_name: str = None, title: str = None): - self.Id = Id - - # All KEYS in the document will be dynamically indexes - # Fields added to the document after index creation time wil also get indexed - self.first_name = first_name - self.last_name = last_name - self.title = title - # ... -`} - - - - - -{`// Sample document content - \{ + + + ```python + class Product: + def __init__( + self, + Id: str = None, + FirstName: str = None, + LastName: str = None, + Title: str = None, + ): + self.Id = Id + + # All KEYS in the document will be dynamically indexed + # Fields added to the document after index creation time will also get indexed + self.FirstName = FirstName + self.LastName = LastName + self.Title = Title + # ... + ``` + + + + ```json + // Sample document content + { "FirstName": "John", "LastName": "Doe", "Title": "Engineer", // ... -\} -`} - - + } + ``` + * **The index**: - The below index will index any field from the document, + + The following index will index any field from the document, a dynamic-index-field will be created for each field. New fields added to the document after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAnyField_JS(AbstractJavaScriptIndexCreationTask): - def __init__(self): - super().__init__() - # This will index EVERY FIELD under the top level of the document - self.maps = { - """ - map('Products', function (p) { - return { - _: Object.keys(p).map(key => createField(key, p[key], - { indexing: 'Search', storage: true, termVector: null })) - } - }) - """ - } -`} - - - + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. + + The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection. + + + + ```python + class Products_ByAnyField_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + # This will index EVERY FIELD under the top level of the document + self.maps = [ + """ + map('Products', function (p) { + return { + _: Object.keys(p).map(key => createField(key, p[key])) + }; + }) + """ + ] + ``` + + * **The query**: - * To get all documents with some 'LastName' use: - - - -{`# 'last_name' is a dynamic-index-field that was indexed from the document -matching_documents = list( - session.query_index_type(Products_ByAnyField_JS, Product).where_equals("last_name", "Doe") -) -`} - - - - -{`// 'LastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where LastName = "Doe" -`} - - - + + To get all documents with a specific _LastName_ use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Products_ByAnyField_JS, Product) + # 'LastName' is a dynamic-index-field that was indexed from the document + .where_equals("LastName", "Doe") + ) + ``` + + + ```sql + // 'LastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where LastName = "Doe" + ``` + + + + + + + + + + + +### Example - basic + + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + + +* **The document**: + + + ```python + class Product: + def __init__(self, Id: str = None, ProductType: str = None, PricePerUnit: int = None): + self.Id = Id + + # The VALUE of ProductType will be dynamically indexed + self.ProductType = ProductType + self.PricePerUnit = PricePerUnit + ``` + + + + ```json + // Sample document content + { + "ProductType": "Electronics", + "PricePerUnit": 23 + } + ``` + + +* **The index**: + + The following index will index the **value** of document field 'ProductType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + + ```python + class Products_ByProductType(AbstractIndexCreationTask): + def __init__(self): + super().__init__() + self.map = ( + "from p in docs.Products select new { " + # Call 'CreateField' to generate the dynamic-index-fields + # The field name will be the value of document field 'ProductType' + # The field terms will be derived from document field 'PricePerUnit' + "_ = CreateField(p.ProductType, p.PricePerUnit) " + "}" + ) + ``` + + + ```python + class Products_ByProductType_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (p) { + return { + // The field name will be the value of document field 'ProductType' + // The field terms will be derived from document field 'PricePerUnit' + _: createField(p.ProductType, p.PricePerUnit) + }; + }) + """ + ] + ``` + + + +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Products_ByProductType, Product) + # 'Electronics' is the dynamic-index-field that was indexed + # from document field 'ProductType' + .where_equals("Electronics", 23) + ) + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'ProductType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + + +### Example - list + + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. + + +* **The document**: + + + ```python + class Attribute: + def __init__(self, PropName: str = None, PropValue: str = None): + self.PropName = PropName + self.PropValue = PropValue + + + class Product: + def __init__(self, Id: str = None, Name: str = None, Attributes: List[Attribute] = None): + self.Id = Id + self.Name = Name + + # For each element in this list, + # the VALUE of property 'PropName' will be dynamically indexed. + # e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + self.Attributes = Attributes + ``` + + + + ```json + // Sample document content + { + "Name": "SomeName", + "Attributes": [ + { + "PropName": "Color", + "PropValue": "Blue" + }, + { + "PropName": "Width", + "PropValue": "10" + }, + { + "PropName": "Length", + "PropValue": "20" + }, + // ... + ] + } + ``` + + +* **The index**: + + The following index will create a dynamic-index-field per item in the document's `Attributes` list. + New items added to the Attributes list after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the item's PropName **value**. + E.g., 'PropName' value `Width` will be a dynamic-index-field. + + + + ```python + class Attributes_ByName(AbstractIndexCreationTask): + def __init__(self): + super().__init__() + self.map = ( + "from a in docs.Products select new { " + # Define the dynamic-index-fields by calling 'CreateField' + # A dynamic-index-field will be generated for each item in 'Attributes' + + # For each item, the field name will be the value of field 'PropName' + # The field terms will be derived from field 'PropValue' + "_ = a.Attributes.Select(item => CreateField(item.PropName, item.PropValue)), " + + # A regular index field can be defined as well: + "Name = a.Name " + "}" + ) + ``` + + + ```python + class Attributes_ByName_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (p) { + return { + // For each item, + // the field name will be the value of field 'PropName' + // The field terms will be derived from field 'PropValue' + _: p.Attributes.map(item => createField(item.PropName, item.PropValue)), + + // A regular index field can be defined as well: + Name: p.Name + }; + }) + """ + ] + ``` + + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Attributes_ByName, Product) + # 'Width' is a dynamic-index-field that was indexed from the Attributes list + .where_equals("Width", 10) + ) + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the Attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `FieldIndexing.Search` in `CreateFieldOptions`. +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). -## Indexing documents fields VALUES +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. -## Example - basic + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. - + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```python + class Product: + def __init__(self, Id: str = None, Descriptions: Dict[str, str] = None): + self.Id = Id + self.Descriptions = Descriptions + ``` + + + + ```json + // Sample document content + { + "Descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + -* **The document**: - - -{`class Product: - def __init__(self, Id: str = None, product_type: str = None, price_per_unit: float = None): - self.Id = Id - - # The VALUE of ProductType will be dynamically indexed - self.product_type = product_type - self.price_per_unit = price_per_unit -`} - - +* **The index**: - - -{`// Sample document content -\{ - "ProductType": "Electronics", - "PricePerUnit": 23 -\} -`} - - + The following index creates one dynamic field for each entry in the `Descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. -* **The index**: - The below index will index the **value** of document field 'ProductType'. - - This value will be the dynamic-index-field name on which you can query. - E.g., Field value `Electronics` will be the dynamic-index-field. + Each generated dynamic field is indexed for **full-text search**. + + + + ```python + class Products_ByLocalizedDescription(AbstractIndexCreationTask): + USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = ( + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery" + ) + + def __init__(self): + super().__init__() + self.map = ( + "from product in docs.Products select new { " + " _ = product.Descriptions.Select(x => " + " CreateField(" + " \"Description_\" + x.Key, " + " x.Value, " + # Index each generated dynamic field for FTS + " new CreateFieldOptions { " + " Indexing = FieldIndexing.Search, " + " Storage = FieldStorage.No " + " })) " + "}" + ) + + self._store_all_fields(FieldStorage.NO) + + # Set this configuration key to true so that search() queries on dynamic fields + # will use the 'Default Search Analyzer' when the generated field name + # has no explicit analyzer configuration. + self.configuration = { + Products_ByLocalizedDescription.USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS: "true" + } + ``` + + + ```python + class Products_ByLocalizedDescription_JS(AbstractJavaScriptIndexCreationTask): + USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = ( + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery" + ) + + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (product) { + return { + _: Object.keys(product.Descriptions).map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: 'Search', + storage: 'No' + }); + }) + }; + }) + """ + ] + + # Apply a storage default to every generated field + self.fields = { + constants.Documents.Indexing.Fields.ALL_FIELDS: + IndexFieldOptions(storage=FieldStorage.NO) + } + + # Set this configuration key to true so that search() queries on dynamic fields + # will use the 'Default Search Analyzer' when the generated field name + # has no explicit analyzer configuration. + self.configuration = { + Products_ByLocalizedDescription_JS.USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS: "true" + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```python + results = list( + session.advanced + .document_query_from_index_type(Products_ByLocalizedDescription, Product) + .search("Description_English", "north wind") + ) + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + + +--- + + + +### Configuring analyzers for dynamic fields + +`CreateFieldOptions` does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field - -{`class Products_ByProductType(AbstractIndexCreationTask): +```python +class Products_ByLocalizedDescription(AbstractIndexCreationTask): def __init__(self): super().__init__() + self.map = ( + "from product in docs.Products select new { " + " _ = product.Descriptions.Select(x => " + " CreateField(" + " \"Description_\" + x.Key, " + " x.Value, " + " new CreateFieldOptions { " + " Indexing = FieldIndexing.Search, " + " Storage = FieldStorage.No " + " })) " + "}" + ) - # Call 'CreateField' to generate the dynamic-index-fields - # The field name will be the value of document field 'product_type' - # The field terms will be derived from document field 'price_per_unit' - self.map = "from p in docs.Products select new { _ = CreateField(p.product_type, p.price_per_unit)}" -`} - + self._store_all_fields(FieldStorage.NO) + + # Configure an analyzer for a known generated dynamic field name + # ============================================================== + self._analyze("Description_English", "StopAnalyzer") +``` - - -{`class Products_ByProductType_JS(AbstractJavaScriptIndexCreationTask): + +```python +class Products_ByLocalizedDescription_JS(AbstractJavaScriptIndexCreationTask): def __init__(self): super().__init__() - self.maps = { + self.maps = [ """ - map('Products', function (p) { + map('Products', function (product) { return { - _: createField(p.product_type, p.price_per_unit, - { indexing: 'Search', storage: true, termVector: null }) + _: Object.keys(product.Descriptions).map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; }) """ + ] + + # Configure an analyzer for a known generated dynamic field name + # ============================================================== + self.fields = { + constants.Documents.Indexing.Fields.ALL_FIELDS: + IndexFieldOptions(storage=FieldStorage.NO), + "Description_English": + IndexFieldOptions(analyzer="StopAnalyzer") } -`} - - - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`# 'electronics' is the dynamic-index-field that was indexed from the document 'product_type' -matching_documents = list( - session.advanced.document_query_from_index_type(Products_ByProductType, Product).where_equals( - "electronics", 23 - ) -) -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -from index 'Products/ByProductType' where Electronics = 23 -`} - +``` -## Example - list - - - -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. - - - -* **The document**: - - -{`class Attribute: - def __init__(self, prop_name: str = None, prop_value: str = None): - self.prop_name = prop_name - self.prop_value = prop_value - - -class Product: - def __init__(self, Id: str = None, name: str = None, attributes: List[Attribute] = None): - self.Id = Id - self.name = name - # For each element in this list, the VALUE of property 'prop_name' will be dynamically indexed - # e.g. color, width, length (in ex. below) will become dynamic-index-field - self.attributes = attributes -`} - - - - - -{`// Sample document content -\{ -Name": "SomeName", -Attributes": [ - \{ - "PropName": "Color", - "PropValue": "Blue" - \}, - \{ - "PropName": "Width", - "PropValue": "10" - \}, - \{ - "PropName": "Length", - "PropValue": "20" - \}, - ... - -\} -`} - - - -* **The index**: - The below index will create a dynamic-index-field per item in the document's `Attributes` list. - New items added to the Attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's PropName **value**. - E.g., 'PropName' value `width` will be a dynamic-index-field. +#### Configure a fallback analyzer for all fields - -{`class Attributes_ByName(AbstractIndexCreationTask): +```python +class Products_ByLocalizedDescription(AbstractIndexCreationTask): def __init__(self): super().__init__() self.map = ( - "from a in docs.Products select new " - "{ _ = a.attributes.Select( item => CreateField(item.prop_name, item.prop_value)), name = a.name " + "from product in docs.Products select new { " + " _ = product.Descriptions.Select(x => " + " CreateField(" + " \"Description_\" + x.Key, " + " x.Value, " + " new CreateFieldOptions { " + " Indexing = FieldIndexing.Search, " + " Storage = FieldStorage.No " + " })) " "}" ) -`} - + + self._store_all_fields(FieldStorage.NO) + + # Use this as a fallback for fields that do not have + # their own explicit analyzer configuration + # ================================================== + self._analyze(constants.Documents.Indexing.Fields.ALL_FIELDS, "StopAnalyzer") +``` - - -{`class Attributes_ByName_JS(AbstractJavaScriptIndexCreationTask): + +```python +class Products_ByLocalizedDescription_JS(AbstractJavaScriptIndexCreationTask): def __init__(self): super().__init__() - self.maps = { + self.maps = [ """ - map('Products', function (p) { + map('Products', function (product) { return { - _: p.Attributes.map(item => createField(item.PropName, item.PropValue, - { indexing: 'Search', storage: true, termVector: null })), - Name: p.Name + _: Object.keys(product.Descriptions).map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; }) """ + ] + + # Use this as a fallback for fields that do not have + # their own explicit analyzer configuration + # ================================================== + self.fields = { + constants.Documents.Indexing.Fields.ALL_FIELDS: + IndexFieldOptions(storage=FieldStorage.NO, analyzer="StopAnalyzer") } -`} - - - - -* **The query**: - To get all documents matching a specific attribute property use: - - - -{`matching_documents = list( - session.advanced.document_query_from_index_type(Attributes_ByName, Product).where_equals( - "width", 10 - ) -) -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the Attributes list -from index 'Attributes/ByName' where Width = 10 -`} - +``` + + + + +### Which analyzer is used for the query term? + +The following applies when a `search()` query targets a **dynamic field**: + +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `self._analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `self._analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | + + + -## CreateField syntax + #### Syntax for Index: - - -{`object CreateField(string name, object value); + +```python +CreateField(name, value) -object CreateField(string name, object value, bool stored, bool analyzed); +CreateField(name, value, stored, analyzed) -object CreateField(string name, object value, CreateFieldOptions options); -`} - +CreateField(name, value, CreateFieldOptions) +``` #### Syntax for JavaScript-index: - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` -| Parameters | Type | Description | -|------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **name** | `string` | Name of the dynamic-index-field | -| **value** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`False` - will set `FieldStorage.NO` (default value)
`True` - will set `FieldStorate.YES` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`None` - `FieldIndexing.Default` (default value)
`False` - `FieldIndexing.Exact`
`True` - `FieldIndexing.Search` | -| **options** | `CreateFieldOptions` | Dynamic-index-field options | +| Parameters | Type | Description | +|----------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorage.Yes` | +| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | +| **options** | `CreateFieldOptions` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| | **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -496,17 +892,15 @@ object CreateField(string name, object value, CreateFieldOptions options); + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/versioned_docs/version-7.0/indexes/querying/content/_searching-csharp.mdx b/versioned_docs/version-7.0/indexes/querying/content/_searching-csharp.mdx index d20ef5a65e..f0be0e6ffd 100644 --- a/versioned_docs/version-7.0/indexes/querying/content/_searching-csharp.mdx +++ b/versioned_docs/version-7.0/indexes/querying/content/_searching-csharp.mdx @@ -16,11 +16,13 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) * [Searching with wildcards](../../../indexes/querying/searching.mdx#searching-with-wildcards) * [When using RavenStandardAnalyzer or StandardAnalyzer or NGramAnalyzer](../../../indexes/querying/searching.mdx#when-usingoror) @@ -28,6 +30,7 @@ import CodeBlock from '@theme/CodeBlock'; * [When using the Exact analyzer](../../../indexes/querying/searching.mdx#when-using-the-exact-analyzer) + ## Indexing single field for FTS #### The index: @@ -262,7 +265,7 @@ where (search(EmployeeData, "Manager") or search(EmployeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method in the _Map_ function to extract all property values and index them in a single searchable field. * This approach makes the index robust to changes in the document schema. @@ -377,7 +380,14 @@ where search(AllValues, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). ## Boosting search results diff --git a/versioned_docs/version-7.0/indexes/querying/content/_searching-java.mdx b/versioned_docs/version-7.0/indexes/querying/content/_searching-java.mdx index 7d85246ced..4accf3f512 100644 --- a/versioned_docs/version-7.0/indexes/querying/content/_searching-java.mdx +++ b/versioned_docs/version-7.0/indexes/querying/content/_searching-java.mdx @@ -388,4 +388,11 @@ where search(allValues, "tofu") +## Indexing dynamic fields for FTS +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). \ No newline at end of file diff --git a/versioned_docs/version-7.0/indexes/querying/content/_searching-nodejs.mdx b/versioned_docs/version-7.0/indexes/querying/content/_searching-nodejs.mdx index 4ca66b5877..cb73bfd7c0 100644 --- a/versioned_docs/version-7.0/indexes/querying/content/_searching-nodejs.mdx +++ b/versioned_docs/version-7.0/indexes/querying/content/_searching-nodejs.mdx @@ -5,7 +5,7 @@ import CodeBlock from '@theme/CodeBlock'; -* Prior to reading this article, please refer to [full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) +* Prior to reading this article, please refer to [Full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) to learn about the `search` method. * **All capabilities** provided by `search` with a dynamic query can also be used when querying a static-index. @@ -17,11 +17,13 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) * [Searching with wildcards](../../../indexes/querying/searching.mdx#searching-with-wildcards) * [When using RavenStandardAnalyzer or StandardAnalyzer or NGramAnalyzer](../../../indexes/querying/searching.mdx#when-usingoror) @@ -29,6 +31,7 @@ import CodeBlock from '@theme/CodeBlock'; * [When using the Exact analyzer](../../../indexes/querying/searching.mdx#when-using-the-exact-analyzer) + ## Indexing single field for FTS #### The index: @@ -163,7 +166,7 @@ where (search(employeeData, "Manager") or search(employeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, which is available when using **a C# LINQ string** that is assigned to the `map` property in the Node.js index class, as shown in the example below. @@ -229,6 +232,14 @@ where search(AllValues, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). ## Boosting search results diff --git a/versioned_docs/version-7.0/indexes/querying/content/_searching-php.mdx b/versioned_docs/version-7.0/indexes/querying/content/_searching-php.mdx index 55261ce19f..510a6f8752 100644 --- a/versioned_docs/version-7.0/indexes/querying/content/_searching-php.mdx +++ b/versioned_docs/version-7.0/indexes/querying/content/_searching-php.mdx @@ -5,7 +5,7 @@ import CodeBlock from '@theme/CodeBlock'; -* Prior to reading this article, please refer to [full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) +* Prior to reading this article, please refer to [Full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) to learn about the `search` method. * **All capabilities** provided by `search` with a dynamic query can also be used when querying a static-index. @@ -17,14 +17,17 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) + ## Indexing single field for FTS #### The index: @@ -253,7 +256,7 @@ where (search(EmployeeData, "Manager") or search(EmployeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, which is available in the **C# LINQ string** that is assigned to the `map` property in the PHP index class, as shown in the example below. @@ -332,6 +335,15 @@ where search(allValues, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). + ## Boosting search results diff --git a/versioned_docs/version-7.0/indexes/querying/content/_searching-python.mdx b/versioned_docs/version-7.0/indexes/querying/content/_searching-python.mdx index f06c29d94f..88c6aa452f 100644 --- a/versioned_docs/version-7.0/indexes/querying/content/_searching-python.mdx +++ b/versioned_docs/version-7.0/indexes/querying/content/_searching-python.mdx @@ -5,7 +5,7 @@ import CodeBlock from '@theme/CodeBlock'; -* Prior to reading this article, please refer to [full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) +* Prior to reading this article, please refer to [Full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) to learn about the `search` method. * **All capabilities** provided by `search` with a dynamic query can also be used when querying a static-index. @@ -17,14 +17,17 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) + ## Indexing single field for FTS #### The index: @@ -167,7 +170,7 @@ where (search(EmployeeData, "Manager") or search(EmployeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, which is available in the **C# LINQ string** that is assigned to the `map` property in the Python index class, as shown in the example below. @@ -238,6 +241,14 @@ where search(all_values, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). ## Boosting search results diff --git a/versioned_docs/version-7.0/indexes/using-dynamic-fields.mdx b/versioned_docs/version-7.0/indexes/using-dynamic-fields.mdx index e9ffb33a07..e465ba5431 100644 --- a/versioned_docs/version-7.0/indexes/using-dynamic-fields.mdx +++ b/versioned_docs/version-7.0/indexes/using-dynamic-fields.mdx @@ -1,7 +1,8 @@ --- -title: "Indexes: Dynamic Index Fields" -sidebar_label: Dynamic Fields -sidebar_position: 28 +title: "Dynamic Index Fields" +sidebar_label: "Dynamic Index Fields" +description: "Index document fields dynamically in RavenDB when field names are unknown at index definition time, using the CreateField method." +sidebar_position: 27 supported_languages: ["csharp", "java", "python", "php", "nodejs"] see_also: - title: "Boosting" @@ -31,7 +32,6 @@ import UsingDynamicFieldsPython from './content/_using-dynamic-fields-python.mdx import UsingDynamicFieldsPhp from './content/_using-dynamic-fields-php.mdx'; import UsingDynamicFieldsNodejs from './content/_using-dynamic-fields-nodejs.mdx'; - diff --git a/versioned_docs/version-7.0/server/configuration/indexing-configuration.mdx b/versioned_docs/version-7.0/server/configuration/indexing-configuration.mdx index a3f9fce727..7bde291911 100644 --- a/versioned_docs/version-7.0/server/configuration/indexing-configuration.mdx +++ b/versioned_docs/version-7.0/server/configuration/indexing-configuration.mdx @@ -102,6 +102,7 @@ import LanguageContent from "@site/src/components/LanguageContent"; [Indexing.OrderByTicksAutomaticallyWhenDatesAreInvolved](../../server/configuration/indexing-configuration.mdx#indexingorderbyticksautomaticallywhendatesareinvolved) [Indexing.QueryClauseCache.Disabled](../../server/configuration/indexing-configuration.mdx#indexingqueryclausecachedisabled) [Indexing.QueryClauseCache.RepeatedQueriesTimeFrameInSec](../../server/configuration/indexing-configuration.mdx#indexingqueryclausecacherepeatedqueriestimeframeinsec) + [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) [Indexing.ScratchSpaceLimitInMb](../../server/configuration/indexing-configuration.mdx#indexingscratchspacelimitinmb) [Indexing.Static.SearchEngineType](../../server/configuration/indexing-configuration.mdx#indexingstaticsearchenginetype) [Indexing.Throttling.TimeIntervalInMs](../../server/configuration/indexing-configuration.mdx#indexingthrottlingtimeintervalinms) @@ -988,6 +989,33 @@ Queries that repeat within this time frame will be considered worth caching. +## Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery + +* This setting controls which analyzer is used for the **query term** when a **`search()` query** targets a [Dynamic index field](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + that was indexed for full-text search, and no analyzer is explicitly configured for that field in the index definition. + + * `false` (default): + The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) is used to analyze the **query term**. + This analyzer can be configured via the [Indexing.Analyzers.Default](../../server/configuration/indexing-configuration.mdx#indexinganalyzersdefault) configuration key. + + * `true`: + The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) is used to analyze the **query term**. + This analyzer can be configured via the [Indexing.Analyzers.Search.Default](../../server/configuration/indexing-configuration.mdx#indexinganalyzerssearchdefault) configuration key. + +* See the table in [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + for the complete analyzer-selection rules based on this configuration key and the analyzer configured for the dynamic index field. + +* This configuration applies to both the Lucene and Corax search engines. + The default is `false` to preserve backward compatibility. + +--- + +- **Type**: `bool` +- **Default**: `false` +- **Scope**: Server-wide, or per database, or per index + + + ## Indexing.ScratchSpaceLimitInMb * Amount of scratch space in megabytes that we allow to use for the index storage. diff --git a/versioned_docs/version-7.1/indexes/assets/dynamic-index-fields-1.png b/versioned_docs/version-7.1/indexes/assets/dynamic-index-fields-1.png index b13d9076f8..a20adda0ec 100644 Binary files a/versioned_docs/version-7.1/indexes/assets/dynamic-index-fields-1.png and b/versioned_docs/version-7.1/indexes/assets/dynamic-index-fields-1.png differ diff --git a/versioned_docs/version-7.1/indexes/assets/dynamic-index-fields-2.png b/versioned_docs/version-7.1/indexes/assets/dynamic-index-fields-2.png index dd2b4082ea..c5b1bc14bc 100644 Binary files a/versioned_docs/version-7.1/indexes/assets/dynamic-index-fields-2.png and b/versioned_docs/version-7.1/indexes/assets/dynamic-index-fields-2.png differ diff --git a/versioned_docs/version-7.1/indexes/content/_using-analyzers-csharp.mdx b/versioned_docs/version-7.1/indexes/content/_using-analyzers-csharp.mdx index 6022cfd145..2e824fea4b 100644 --- a/versioned_docs/version-7.1/indexes/content/_using-analyzers-csharp.mdx +++ b/versioned_docs/version-7.1/indexes/content/_using-analyzers-csharp.mdx @@ -24,7 +24,7 @@ import CodeBlock from '@theme/CodeBlock'; * [Understanding the role of analyzers](../../indexes/using-analyzers.mdx#understanding-the-role-of-analyzers) * [Analyzers available in RavenDB](../../indexes/using-analyzers.mdx#analyzers-available-in-ravendb) * [Setting analyzer for index-field](../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field) - * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendb) + * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendbs-default-analyzers) * [Disabling indexing for index-field](../../indexes/using-analyzers.mdx#disabling-indexing-for-index-field) * [Creating custom analyzers](../../indexes/using-analyzers.mdx#creating-custom-analyzers) * [Viewing the indexed terms](../../indexes/using-analyzers.mdx#viewing-the-indexed-terms) diff --git a/versioned_docs/version-7.1/indexes/content/_using-analyzers-nodejs.mdx b/versioned_docs/version-7.1/indexes/content/_using-analyzers-nodejs.mdx index 6afd3b6202..fae2cdd2b9 100644 --- a/versioned_docs/version-7.1/indexes/content/_using-analyzers-nodejs.mdx +++ b/versioned_docs/version-7.1/indexes/content/_using-analyzers-nodejs.mdx @@ -24,7 +24,7 @@ import CodeBlock from '@theme/CodeBlock'; * [Understanding the role of analyzers](../../indexes/using-analyzers.mdx#understanding-the-role-of-analyzers) * [Analyzers available in RavenDB](../../indexes/using-analyzers.mdx#analyzers-available-in-ravendb) * [Setting analyzer for index-field](../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field) - * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendb) + * [RavenDB's default analyzers](../../indexes/using-analyzers.mdx#ravendbs-default-analyzers) * [Disabling indexing for index-field](../../indexes/using-analyzers.mdx#disabling-indexing-for-index-field) * [Creating custom analyzers](../../indexes/using-analyzers.mdx#creating-custom-analyzers) * [Viewing the indexed terms](../../indexes/using-analyzers.mdx#viewing-the-indexed-terms) diff --git a/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-csharp.mdx b/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-csharp.mdx index b4c4f518df..5bf052d482 100644 --- a/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-csharp.mdx +++ b/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-csharp.mdx @@ -2,533 +2,923 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. - -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. - -* In this page: - - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -## Indexing documents fields KEYS - -## Example - index any field under object - - + -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + + +### Example - index every field under an object + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. * **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - - // The KEYS under the Attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - public Dictionary Attributes \{ get; set; \} -\} -`} - - - - -{`// Sample document content -\{ - "Attributes": \{ - "Color": "Red", - "Size": 42 - \} -\} -`} - - + + + ```csharp + public class Product + { + public string Id { get; set; } + + // The KEYS under the Attributes object will be dynamically indexed + // Fields added to this object after index creation time will also get indexed + public Dictionary Attributes { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "Attributes": { + "Color": "Red", + "Size": 42 + } + } + ``` + * **The index**: - The below index will index any field under the `Attributes` object from the document, + + The following index will index any field under the `Attributes` object from the document, a dynamic-index-field will be created for each such field. New fields added to the object after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. - - - -{`public class Products_ByAttributeKey : AbstractIndexCreationTask -{ - public Products_ByAttributeKey() + + + ```csharp + public class Products_ByAttributeKey : AbstractIndexCreationTask { - Map = products => from p in products - select new - { - // Call 'CreateField' to generate dynamic-index-fields from the Attributes object keys - // Using '_' is just a convention. Any other string can be used instead of '_' - - // The actual field name will be item.Key - // The actual field terms will be derived from item.Value - _ = p.Attributes.Select(item => CreateField(item.Key, item.Value)) - }; + public Products_ByAttributeKey() + { + Map = products => from p in products + select new + { + // Call 'CreateField' to generate dynamic-index-fields + // from the Attributes object keys. + + // Using '_' is just a convention. + // Any other string can be used instead of '_' + + // The actual field name will be 'item.Key' + // The actual field terms will be derived from 'item.Value' + _ = p.Attributes.Select(item => CreateField(item.Key, item.Value)) + }; + } } -} -`} - - - - -{`public class Products_ByAttributeKey_JS : AbstractJavaScriptIndexCreationTask -{ - public Products_ByAttributeKey_JS() + ``` + + + ```csharp + public class Products_ByAttributeKey_JS : AbstractJavaScriptIndexCreationTask { - Maps = new HashSet + public Products_ByAttributeKey_JS() { - @"map('Products', function (p) { - return { - _: Object.keys(p.Attributes).map(key => createField(key, p.Attributes[key], - { indexing: 'Search', storage: true, termVector: null })) - }; - })" - }; + Maps = new HashSet + { + @"map('Products', function (p) { + return { + // The field name will be the key + // The field terms will be derived from the corresponding value + _: Object.keys(p.Attributes) + .map(key => createField(key, p.Attributes[key])) + }; + })" + }; + } } -} -`} - - - - + ``` + + * **The query**: - * You can now query the generated dynamic-index fields. - The `_` property is Not queryable but used only in the index definition syntax. - * To get all documents with some 'Size' use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'Size' is a dynamic-index-field that was indexed from the Attributes object - .WhereEquals("Size", 42) - .ToList(); -`} - - - - -{`// 'Size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey' where Size = 42 -`} - - - + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _Size_ use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'Size' is a dynamic-index-field that was indexed from the Attributes object + .WhereEquals("Size", 42) + .ToList(); + ``` + + + ```sql + // 'Size' is a dynamic-index-field that was indexed from the Attributes object + from index 'Products/ByAttributeKey' where Size = 42 + ``` + + + + + Dynamic index fields are queried with _DocumentQuery_ or _RQL_ because the field names (e.g. `Size`, `Color`) + are generated at indexing time and are not properties on the _Product_ class. + + A strongly typed LINQ query such as `Query().Where(p => p.Size == 42)` would not compile. + Use the string-based `WhereEquals("Size", 42)` or RQL `where Size = 42` instead. + -## Example - index any field + - + + +### Example - index every field + +The following example allows you to: * Define an index on a collection **without** needing any common structure between the indexed documents. - * After index is deployed, any new field added to the document will be indexed as well. - + * After the index is deployed, any new field added to any document in the collection will be indexed as well. - + -Consider whether this is really necessary, as indexing every single field can end up costing time and disk space. +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - public string FirstName \{ get; set; \} - public string LastName \{ get; set; \} - public string Title \{ get; set; \} - // ... -\} -`} - - - - - -{`// Sample document content - \{ - "FirstName": "John", - "LastName": "Doe", - "Title": "Engineer", + + + ```csharp + public class Product + { + public string Id { get; set; } + + // All KEYS in the document will be dynamically indexed + // Fields added to the document after index creation time will also get indexed + public string FirstName { get; set; } + public string LastName { get; set; } + public string Title { get; set; } // ... -\} -`} - - + } + ``` + + + + ```json + // Sample document content + { + "FirstName": "John", + "LastName": "Doe", + "Title": "Engineer", + // ... + } + ``` + * **The index**: - The below index will index any field from the document, + + The following index will index any field from the document, a dynamic-index-field will be created for each field. New fields added to the document after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. + + The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection. - - - -{`public class Products_ByAnyField_JS : AbstractJavaScriptIndexCreationTask -{ - public Products_ByAnyField_JS() + + + ```csharp + public class Products_ByAnyField_JS : AbstractJavaScriptIndexCreationTask { - // This will index EVERY FIELD under the top level of the document - Maps = new HashSet + public Products_ByAnyField_JS() { - @"map('Products', function (p) { - return { - _: Object.keys(p).map(key => createField(key, p[key], - { indexing: 'Search', storage: true, termVector: null })) - } - })" - }; + // This will index EVERY FIELD under the top level of the document + Maps = new HashSet + { + @"map('Products', function (p) { + return { + _: Object.keys(p).map(key => createField(key, p[key])) + }; + })" + }; + } } -} -`} - - - + ``` + + * **The query**: - * To get all documents with some 'LastName' use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'LastName' is a dynamic-index-field that was indexed from the document - .WhereEquals("LastName", "Doe") - .ToList(); -`} - - - - -{`// 'LastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where LastName = "Doe" -`} - - - + + To get all documents with a specific _LastName_ use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'LastName' is a dynamic-index-field that was indexed from the document + .WhereEquals("LastName", "Doe") + .ToList(); + ``` + + + ```sql + // 'LastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where LastName = "Doe" + ``` + + + + + + + + + +### Example - basic -## Indexing documents fields VALUES + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + -## Example - basic +* **The document**: + + + ```csharp + public class Product + { + public string Id { get; set; } + + // The VALUE of ProductType will be dynamically indexed + public string ProductType { get; set; } + public int PricePerUnit { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "ProductType": "Electronics", + "PricePerUnit": 23 + } + ``` + - +* **The index**: + + The following index will index the **value** of document field 'ProductType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + + ```csharp + public class Products_ByProductType : AbstractIndexCreationTask + { + public Products_ByProductType() + { + Map = products => from p in products + select new + { + // Call 'CreateField' to generate the dynamic-index-fields + // The field name will be the value of document field 'ProductType' + // The field terms will be derived from document field 'PricePerUnit' + _ = CreateField(p.ProductType, p.PricePerUnit) + }; + } + } + ``` + + + ```csharp + public class Products_ByProductType_JS : AbstractJavaScriptIndexCreationTask + { + public Products_ByProductType_JS() + { + Maps = new HashSet + { + @"map('Products', function (p) { + return { + // The field name will be the value of document field 'ProductType' + // The field terms will be derived from document field 'PricePerUnit' + _: createField(p.ProductType, p.PricePerUnit) + }; + })" + }; + } + } + ``` + + + +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'ProductType' + .WhereEquals("Electronics", 23) + .ToList(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'ProductType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. -* For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---index-a-list-of-properties) below. +### Example - list + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. * **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - - // The VALUE of ProductType will be dynamically indexed - public string ProductType \{ get; set; \} - public int PricePerUnit \{ get; set; \} -\} -`} - - - - - -{`// Sample document content -\{ - "ProductType": "Electronics", - "PricePerUnit": 23 -\} -`} - - + + + ```csharp + public class Product + { + public string Id { get; set; } + public string Name { get; set; } + + // For each element in this list, + // the VALUE of property 'PropName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + public List Attributes { get; set; } + } + + public class Attribute + { + public string PropName { get; set; } + public string PropValue { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "Name": "SomeName", + "Attributes": [ + { + "PropName": "Color", + "PropValue": "Blue" + }, + { + "PropName": "Width", + "PropValue": "10" + }, + { + "PropName": "Length", + "PropValue": "20" + }, + ... + ] + } + ``` + * **The index**: - The below index will index the **value** of document field 'ProductType'. + + The following index will create a dynamic-index-field per item in the document's `Attributes` list. + New items added to the Attributes list after index creation time will be dynamically indexed as well. - This value will be the dynamic-index-field name on which you can query. - E.g., Field value `Electronics` will be the dynamic-index-field. + The actual dynamic-index-field name on which you can query will be the item's PropName **value**. + E.g., 'PropName' value `Width` will be a dynamic-index-field. + + + + ```csharp + public class Attributes_ByName : AbstractIndexCreationTask + { + public Attributes_ByName() + { + Map = products => from a in products + select new + { + // Define the dynamic-index-fields by calling 'CreateField' + // A dynamic-index-field will be generated for each item in 'Attributes' + + // For each item, the field name will be the value of field 'PropName' + // The field terms will be derived from field 'PropValue' + _ = a.Attributes.Select(item => + CreateField(item.PropName, item.PropValue)), + + // A regular index field can be defined as well: + Name = a.Name + }; + } + } + ``` + + + ```csharp + public class Attributes_ByName_JS : AbstractJavaScriptIndexCreationTask + { + public Attributes_ByName_JS() + { + Maps = new HashSet + { + @"map('Products', function (p) { + return { + // For each item, + // the field name will be the value of field 'PropName' + // The field terms will be derived from field 'PropValue' + _: p.Attributes.map(item => createField( + item.PropName, item.PropValue)), + + // A regular index field can be defined as well: + Name: p.Name + }; + })" + }; + } + } + ``` + + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```csharp + IList matchingDocuments = session + .Advanced + .DocumentQuery() + // 'Width' is a dynamic-index-field that was indexed from the Attributes list + .WhereEquals("Width", 10) + .ToList(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the Attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `FieldIndexing.Search` in `CreateFieldOptions`. + +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `AllFields` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `AllFields` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. + + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. + + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + + + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```csharp + public class Product + { + public string Id { get; set; } + + public Dictionary Descriptions { get; set; } + } + ``` + + + + ```json + // Sample document content + { + "Descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `Descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. + + Each generated dynamic field is indexed for **full-text search**. + + + + ```csharp + public class Products_ByLocalizedDescription : AbstractIndexCreationTask + { + private const string UseSearchAnalyzerForDynamicFields = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription() + { + Map = products => from product in products + select new + { + _ = product.Descriptions.Select(x => + CreateField( + "Description_" + x.Key, + x.Value, + new CreateFieldOptions + { + // Index each generated dynamic field for FTS + Indexing = FieldIndexing.Search, + Storage = FieldStorage.No + })) + }; + + StoreAllFields(FieldStorage.No); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + Configuration[UseSearchAnalyzerForDynamicFields] = "true"; + } + } + ``` + + + ```csharp + public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask + { + private const string UseSearchAnalyzerForDynamicFields = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription_JS() + { + Maps = new HashSet + { + @"map('Products', function (product) { + return { + _: Object.keys(product.Descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: 'Search', + storage: 'No' + }); + }) + }; + })" + }; + + Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions + { + Storage = FieldStorage.No + }; + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + Configuration[UseSearchAnalyzerForDynamicFields] = "true"; + } + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```csharp + List results = session.Advanced + .DocumentQuery() + .Search("Description_English", "north wind") + .ToList(); + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + + +--- + + + +### Configuring analyzers for dynamic fields - - - -{`public class Products_ByProductType : AbstractIndexCreationTask +`CreateFieldOptions` does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `AllFields` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `AllFields` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field + + + +```csharp +public class Products_ByLocalizedDescription : AbstractIndexCreationTask { - public Products_ByProductType() + public Products_ByLocalizedDescription() { - Map = products => from p in products + Map = products => from product in products select new { - // Call 'CreateField' to generate the dynamic-index-fields - // The field name will be the value of document field 'ProductType' - // The field terms will be derived from document field 'PricePerUnit' - _ = CreateField(p.ProductType, p.PricePerUnit) + _ = product.Descriptions.Select(x => + CreateField( + "Description_" + x.Key, + x.Value, + new CreateFieldOptions + { + Indexing = FieldIndexing.Search, + Storage = FieldStorage.No + })) }; + + StoreAllFields(FieldStorage.No); + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + Analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -{`public class Products_ByProductType_JS : AbstractJavaScriptIndexCreationTask + +```csharp +public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask { - public Products_ByProductType_JS() + public Products_ByLocalizedDescription_JS() { Maps = new HashSet { - @"map('Products', function (p) { + @"map('Products', function (product) { return { - _: createField(p.ProductType, p.PricePerUnit, - { indexing: 'Search', storage: true, termVector: null }) + _: Object.keys(product.Descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" }; + + Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions + { + Storage = FieldStorage.No + }; + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + Fields["Description_English"] = new IndexFieldOptions + { + Analyzer = "StopAnalyzer" + }; } } -`} - - - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' - .WhereEquals("Electronics", 23) - .ToList(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -from index 'Products/ByProductType' where Electronics = 23 -`} - +``` -## Example - list - - +#### Configure a fallback analyzer for all fields -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. - - - -* **The document**: - - -{`public class Product -\{ - public string Id \{ get; set; \} - public string Name \{ get; set; \} - - // For each element in this list, the VALUE of property 'PropName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - public List Attributes \{ get; set; \} -\} - -public class Attribute -\{ - public string PropName \{ get; set; \} - public string PropValue \{ get; set; \} -\} -`} - - - - - -{`// Sample document content -\{ -Name": "SomeName", -Attributes": [ - \{ - "PropName": "Color", - "PropValue": "Blue" - \}, - \{ - "PropName": "Width", - "PropValue": "10" - \}, - \{ - "PropName": "Length", - "PropValue": "20" - \}, - ... - -\} -`} - - - -* **The index**: - The below index will create a dynamic-index-field per item in the document's `Attributes` list. - New items added to the Attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's PropName **value**. - E.g., 'PropName' value `Width` will be a dynamic-index-field. - - - - -{`public class Attributes_ByName : AbstractIndexCreationTask + + +```csharp +public class Products_ByLocalizedDescription : AbstractIndexCreationTask { - public Attributes_ByName() + public Products_ByLocalizedDescription() { - Map = products => from a in products + Map = products => from product in products select new { - // Define the dynamic-index-fields by calling CreateField - // A dynamic-index-field will be generated for each item in the Attributes list - - // For each item, the field name will be the value of field 'PropName' - // The field terms will be derived from field 'PropValue' - _ = a.Attributes.Select(item => CreateField(item.PropName, item.PropValue)), - - // A regular index field can be defined as well: - Name = a.Name + _ = product.Descriptions.Select(x => + CreateField( + "Description_" + x.Key, + x.Value, + new CreateFieldOptions + { + Indexing = FieldIndexing.Search, + Storage = FieldStorage.No + })) }; + + StoreAllFields(FieldStorage.No); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + Analyze(Constants.Documents.Indexing.Fields.AllFields, "StopAnalyzer"); } } -`} - +``` - - -{`public class Attributes_ByName_JS : AbstractJavaScriptIndexCreationTask + +```csharp +public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask { - public Attributes_ByName_JS() + public Products_ByLocalizedDescription_JS() { Maps = new HashSet { - @"map('Products', function (p) { + @"map('Products', function (product) { return { - _: p.Attributes.map(item => createField(item.PropName, item.PropValue, - { indexing: 'Search', storage: true, termVector: null })), - Name: p.Name + _: Object.keys(product.Descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" }; + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions + { + Storage = FieldStorage.No, + Analyzer = "StopAnalyzer" + }; } } -`} - - - - -* **The query**: - * To get all documents matching a specific attribute property use: - - - -{`IList matchingDocuments = session - .Advanced - .DocumentQuery() - // 'Width' is a dynamic-index-field that was indexed from the Attributes list - .WhereEquals("Width", 10) - .ToList(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the Attributes list -from index 'Attributes/ByName' where Width = 10 -`} - +``` + + + + +### Which analyzer is used for the query term? + +The following applies when a `search()` query targets a **dynamic field**: + +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `AllFields` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `AllFields` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `Analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `Analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | + -## CreateField syntax + + + #### Syntax for LINQ-index: - - -{`object CreateField(string name, object value); + +```csharp +object CreateField(string name, object value); object CreateField(string name, object value, bool stored, bool analyzed); object CreateField(string name, object value, CreateFieldOptions options); -`} - +``` #### Syntax for JavaScript-index: - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` | Parameters | Type | Description | |----------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| | **fieldName** | `string` | Name of the dynamic-index-field | -| **fieldValue** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorate.Yes` | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorage.Yes` | | **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | | **options** | `CreateFieldOptions` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| | **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -540,11 +930,15 @@ object CreateField(string name, object value, CreateFieldOptions options); -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. + + + +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) + + \ No newline at end of file diff --git a/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-java.mdx b/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-java.mdx index 17f3cbe360..d54ef6bea8 100644 --- a/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-java.mdx +++ b/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-java.mdx @@ -2,479 +2,918 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -* For such dynamic data, you can define indexes with **dynamic-index-fields**. + -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. + -* Any value type can be indexed, string, number, date, etc. + + +### Example - index every field under an object -* An index definition can contain both dynamic-index-fields and regular-index-fields. + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. + -* In this page: +* **The document**: + + + ```java + public class Product { + private String id; - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) + // The KEYS under the attributes object will be dynamically indexed + // Fields added to this object after index creation time will also get indexed + private Map attributes; - + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "attributes": { + "color": "Red", + "size": 42 + } + } + ``` + -## Indexing documents fields KEYS +* **The index**: + + The following index will index any field under the `attributes` object from the document, + a dynamic-index-field will be created for each such field. + New fields added to the object after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `color` & `size` will become the actual dynamic-index-fields. + + + + ```java + public class Products_ByAttributeKey extends AbstractIndexCreationTask { + public Products_ByAttributeKey() { + + // Call 'CreateField' to generate dynamic-index-fields + // from the attributes object keys. + + // Using '_' is just a convention. + // Any other string can be used instead of '_'. + + // The actual field name will be 'item.Key' + // The actual field terms will be derived from 'item.Value' + map = "docs.Products.Select(p => new { " + + " _ = p.attributes.Select(item => " + + " this.CreateField(item.Key, item.Value)) " + + "})"; + } + } + ``` + + + ```java + public class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByAttributeKey_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + // The field name will be the key + // The field terms will be derived from the corresponding value + " _: Object.keys(p.attributes) " + + " .map(key => createField(key, p.attributes[key])) " + + " }; " + + "})" + )); + } + } + ``` + + -## Example - index any field under object +* **The query**: + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _size_ use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Products_ByAttributeKey.class) + // 'size' is a dynamic-index-field that was indexed from the attributes object + .whereEquals("size", 42) + .toList(); + ``` + + + ```sql + // 'size' is a dynamic-index-field that was indexed from the attributes object + from index 'Products/ByAttributeKey' where size = 42 + ``` + + + + + Dynamic index fields are queried with _DocumentQuery_ or _RQL_ because the field names (e.g. `size`, `color`) + are generated at indexing time and are not properties on the _Product_ class. + + Use the string-based `whereEquals("size", 42)` or RQL `where size = 42` instead. + - + -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + + +### Example - index every field + +The following example allows you to: + * Define an index on a collection **without** needing any common structure between the indexed documents. + * After the index is deployed, any new field added to any document in the collection will be indexed as well. + + + +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`public class Product \{ - private String id; - - // The KEYS under the attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - private Dictionary attributes; - - // get + set implementation ... -\} -`} - - - - -{`// Sample document content -\{ - "attributes": \{ - "color": "Red", - "size": 42 - \} -\} -`} - - + + + ```java + public class Product { + private String id; + + // All KEYS in the document will be dynamically indexed + // Fields added to the document after index creation time will also get indexed + private String firstName; + private String lastName; + private String title; + // ... + + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "firstName": "John", + "lastName": "Doe", + "title": "Engineer" + // ... + } + ``` + * **The index**: - The following index will index any field under the `attributes` object from the document, - a dynamic-index-field will be created for each such field. - New fields added to the object after index creation time will be dynamically indexed as well. + + The following index will index any field from the document, + a dynamic-index-field will be created for each field. + New fields added to the document after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + + The example uses an `AbstractJavaScriptIndexCreationTask` because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a map or collection. + + + + ```java + public class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByAnyField_JS() { + + // This will index EVERY FIELD under the top level of the document + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + " _: Object.keys(p).map(key => createField(key, p[key])) " + + " }; " + + "})" + )); + } + } + ``` + + - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - e.g. Keys `color` & `size` will become the actual dynamic-index-fields. +* **The query**: + + To get all documents with a specific _lastName_ use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Products_ByAnyField_JS.class) + // 'lastName' is a dynamic-index-field that was indexed from the document + .whereEquals("lastName", "Doe") + .toList(); + ``` + + + ```sql + // 'lastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where lastName = "Doe" + ``` + + + + + + - - - -{`public class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { - public Products_ByAttributeKey_JS() { - // Call 'createField' to generate dynamic-index-fields from the attributes object keys - // Using '_' is just a convention. Any other string can be used instead of '_' + - // The actual field name will be the key - // The actual field terms will be derived from p.attributes[key] - setMaps(Sets.newHashSet( - "map('Products', function (p) { " + - " return { " + - " _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], " + - " { indexing: 'Search', storage: false, termVector: null })) " + - " }; " + - "}) " - )); - } -} -`} - - - + + +### Example - basic -* **The query**: - * You can now query the generated dynamic-index fields. - Property `_` is Not queryable, it is only used in the index definition syntax. - * To get all documents with some 'size' use: - - - -{`List matchingDocuments = session - .query(Product.class, Products_ByAttributeKey_JS.class) - .whereEquals("size", 42) - .toList(); -`} - - - - -{`// 'size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey/JS' where size = 42 -`} - - - + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + -## Example - index any field +* **The document**: + + + ```java + public class Product { + private String id; - + // The VALUE of productType will be dynamically indexed + private String productType; + private int pricePerUnit; -* Define an index on a collection **without** needing any common structure between the indexed documents. -* After index is deployed, any new field added to the document will be indexed as well. + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "productType": "Electronics", + "pricePerUnit": 23 + } + ``` + - +* **The index**: + + The following index will index the **value** of document field 'productType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + + ```java + public class Products_ByProductType extends AbstractIndexCreationTask { + public Products_ByProductType() { + + // Call 'CreateField' to generate the dynamic-index-field. + // The field name will be the value of document field 'productType'. + // The field terms will be derived from document field 'pricePerUnit'. + map = "docs.Products.Select(p => new { " + + " _ = this.CreateField(p.productType, p.pricePerUnit) " + + "})"; + } + } + ``` + + + ```java + public class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByProductType_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + // The field name will be the value of document field 'productType' + // The field terms will be derived from document field 'pricePerUnit' + " _: createField(p.productType, p.pricePerUnit) " + + " }; " + + "})" + )); + } + } + ``` + + - -Consider if that is a true necessity, as indexing every single field can end up costing time and disk space. +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Products_ByProductType.class) + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + .whereEquals("Electronics", 23) + .toList(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + + +### Example - list + + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. * **The document**: - - -{`public class Product \{ - private String id; - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - private String firstName; - private String lastName; - private String title; - // ... - - // get + set implementation ... -\} -`} - - - - - -{`// Sample document content -\{ - "firstName": "John", - "lastName": "Doe", - "title": "Engineer", - // ... -\} -`} - - + + + ```java + public class Product { + private String id; + private String name; + + // For each element in this list, + // the VALUE of property 'propName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + private List attributes; + + // getters and setters ... + } -* **The index**: - The following index will index any field from the document, - a dynamic-index-field will be created for each field. - New fields added to the document after index creation time will be dynamically indexed as well. + public class Attribute { + private String propName; + private String propValue; - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "name": "SomeName", + "attributes": [ + { + "propName": "Color", + "propValue": "Blue" + }, + { + "propName": "Width", + "propValue": "10" + }, + { + "propName": "Length", + "propValue": "20" + } + // ... + ] + } + ``` + - - - -{`public class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { - public Products_ByAnyField_JS() { +* **The index**: - // This will index EVERY FIELD under the top level of the document - setMaps(Sets.newHashSet( - "map('Products', function (p) { " + - " return { " + - " _: Object.keys(p).map(key => createField(key, p[key], " + - " { indexing: 'Search', storage: true, termVector: null })) " + - " }; " + - "}) " - )); + The following index will create a dynamic-index-field per item in the document's `attributes` list. + New items added to the attributes list after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the item's propName **value**. + E.g., 'propName' value `Width` will be a dynamic-index-field. + + + + ```java + public class Attributes_ByName extends AbstractIndexCreationTask { + public Attributes_ByName() { + + // Define the dynamic-index-fields by calling 'CreateField'. + // A dynamic-index-field will be generated for each item in 'attributes'. + + // For each item, the field name will be the value of field 'propName' + // The field terms will be derived from field 'propValue'. + // A regular index field (Name) is defined as well. + map = "docs.Products.Select(p => new { " + + " _ = p.attributes.Select(item => " + + " this.CreateField(item.propName, item.propValue)), " + + " Name = p.name " + + "})"; + } } -} -`} - - - + ``` + + + ```java + public class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask { + public Attributes_ByName_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (p) { " + + " return { " + + // For each item, + // the field name will be the value of field 'propName'. + // The field terms will be derived from field 'propValue'. + " _: p.attributes.map(item => " + + " createField(item.propName, item.propValue)), " + + // A regular index field can be defined as well: + " Name: p.name " + + " }; " + + "})" + )); + } + } + ``` + + * **The query**: - * To get all documents with some 'lastName' use: - - - -{`List matchingDocuments = session - .query(Product.class, Products_ByAnyField_JS.class) - .whereEquals("lastName", "Doe") - .toList(); -`} - - - - -{`// 'lastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where lastName = "Doe" -`} - - - + + To get all documents matching a specific attribute property use: + + + + ```java + List matchingDocuments = session + .advanced() + .documentQuery(Product.class, Attributes_ByName.class) + // 'Width' is a dynamic-index-field that was indexed from the attributes list + .whereEquals("Width", 10) + .toList(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `indexing: 'Search'` in the `createField` options object (`AbstractJavaScriptIndexCreationTask`) +or `FieldIndexing.Search` in the `CreateFieldOptions` (`AbstractIndexCreationTask`). +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. -## Indexing documents fields VALUES + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). -## Example - basic +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. - + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. -* For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---index-a-list-of-properties) below. + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: -* **The document**: - - -{`public class Product \{ - private String id; - - // The VALUE of productType will be dynamically indexed - private String productType; - private int pricePerUnit; - - // get + set implementation ... -\} -`} - - + + ```java + public class Product { + private String id; - - -{`// Sample document content -\{ - "productType": "Electronics", - "pricePerUnit": 23 -\} -`} - - + private Map descriptions; -* **The index**: - The following index will index the **value** of document field 'productType'. + // getters and setters ... + } + ``` + + + + ```json + // Sample document content + { + "descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `descriptions` map. + For example, the key `English` creates the dynamic field `Description_English`. + + Each generated dynamic field is indexed for **full-text search**. + + + + ```java + public class Products_ByLocalizedDescription extends AbstractIndexCreationTask { + + private static final String USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription() { + map = "docs.Products.Select(product => new { " + + " _ = product.descriptions.Select(x => " + + " this.CreateField( " + + " \"Description_\" + x.Key, " + + " x.Value, " + + " new CreateFieldOptions { " + + // Index each generated dynamic field for FTS + " Indexing = FieldIndexing.Search, " + + " Storage = FieldStorage.No " + + " })) " + + "})"; + + storeAllFields(FieldStorage.NO); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + getConfiguration().put(USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS, "true"); + } + } + ``` + + + ```java + public class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + + private static final String USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public Products_ByLocalizedDescription_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (product) { " + + " return { " + + " _: Object.keys(product.descriptions) " + + " .map(function (key) { " + + " return createField( " + + " 'Description_' + key, " + + " product.descriptions[key], " + + " { " + + // Index each generated dynamic field for FTS + " indexing: 'Search', " + + " storage: false " + + " }); " + + " }) " + + " }; " + + "})" + )); + + Map fields = new HashMap<>(); + + IndexFieldOptions allFieldsOptions = new IndexFieldOptions(); + allFieldsOptions.setStorage(FieldStorage.NO); + + fields.put(Constants.Documents.Indexing.Fields.ALL_FIELDS, allFieldsOptions); + setFields(fields); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + getConfiguration().put(USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS, "true"); + } + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```java + List results = session.advanced() + .documentQuery(Product.class, Products_ByLocalizedDescription.class) + .search("Description_English", "north wind") + .toList(); + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + - This value will be the dynamic-index-field name on which you can query. - e.g. Field value `Electronics` will be the dynamic-index-field. +--- - - - -{`public class Products_ByProductType extends AbstractIndexCreationTask { - public Products_ByProductType() { + + +### Configuring analyzers for dynamic fields + +The `CreateFieldOptions` (and the JavaScript `createField` options) do **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: - // The field name will be the value of document field 'productType' - // The field terms will be derived from document field 'pricePerUnit' - map = "docs.Products.Select(p => new { " + - " _ = this.CreateField(p.productType, p.pricePerUnit) " + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field + + + +```java +public class Products_ByLocalizedDescription extends AbstractIndexCreationTask { + public Products_ByLocalizedDescription() { + map = "docs.Products.Select(product => new { " + + " _ = product.descriptions.Select(x => " + + " this.CreateField( " + + " \"Description_\" + x.Key, " + + " x.Value, " + + " new CreateFieldOptions { " + + " Indexing = FieldIndexing.Search, " + + " Storage = FieldStorage.No " + + " })) " + "})"; + + storeAllFields(FieldStorage.NO); + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`List matchingDocuments = session - .query(Product.class, Products_ByProductType.class) - .whereEquals("Electronics", 23) - .toList(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'productType' -from index 'Products/ByProductType' where Electronics = 23 -`} - - - - -## Example - list + +```java +public class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByLocalizedDescription_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (product) { " + + " return { " + + " _: Object.keys(product.descriptions) " + + " .map(function (key) { " + + " return createField( " + + " 'Description_' + key, " + + " product.descriptions[key], " + + " { " + + " indexing: 'Search', " + + " storage: false " + + " }); " + + " }) " + + " }; " + + "})" + )); - + Map fields = new HashMap<>(); -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. + IndexFieldOptions allFieldsOptions = new IndexFieldOptions(); + allFieldsOptions.setStorage(FieldStorage.NO); + fields.put(Constants.Documents.Indexing.Fields.ALL_FIELDS, allFieldsOptions); - + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + IndexFieldOptions englishOptions = new IndexFieldOptions(); + englishOptions.setAnalyzer("StopAnalyzer"); + fields.put("Description_English", englishOptions); -* **The document**: - - -{`public class Product \{ - private String id; - private String name; - - // For each element in this list, the VALUE of property 'propName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - private List attributes; - - // get + set implementation ... -\} - -public class Attribute \{ - private String propName; - private String propValue; - - // get + set implementation ... -\} -`} - + setFields(fields); + } +} +``` + - - -{`// Sample document content -\{ -name": "SomeName", -attributes": [ - \{ - "propName": "Color", - "propValue": "Blue" - \}, - \{ - "propName": "Width", - "propValue": "10" - \}, - \{ - "propName": "Length", - "propValue": "20" - \}, - ... - -\} -`} - - +#### Configure a fallback analyzer for all fields + + + +```java +public class Products_ByLocalizedDescription extends AbstractIndexCreationTask { + public Products_ByLocalizedDescription() { + map = "docs.Products.Select(product => new { " + + " _ = product.descriptions.Select(x => " + + " this.CreateField( " + + " \"Description_\" + x.Key, " + + " x.Value, " + + " new CreateFieldOptions { " + + " Indexing = FieldIndexing.Search, " + + " Storage = FieldStorage.No " + + " })) " + + "})"; -* **The index**: - The following index will create a dynamic-index-field per item in the document's `attributes` list. - New items added to the attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's propName **value**. - E.g., 'propName' value `Width` will be a dynamic-index-field. - - - - -{`public class Attributes_ByName extends AbstractIndexCreationTask { - public Attributes_ByName() { - - // For each attribute item, the field name will be the value of field 'propName' - // The field terms will be derived from field 'propValue' - // A regular-index-field (Name) is defined as well - map = - "docs.Products.Select(p => new { " + - " _ = p.attributes.Select(item => this.CreateField(item.propName, item.propValue)), " + - " Name = p.name " + - "})"; + storeAllFields(FieldStorage.NO); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + analyze(Constants.Documents.Indexing.Fields.ALL_FIELDS, "StopAnalyzer"); } } -`} - +``` - + +```java +public class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + public Products_ByLocalizedDescription_JS() { + setMaps(Sets.newHashSet( + "map('Products', function (product) { " + + " return { " + + " _: Object.keys(product.descriptions) " + + " .map(function (key) { " + + " return createField( " + + " 'Description_' + key, " + + " product.descriptions[key], " + + " { " + + " indexing: 'Search', " + + " storage: false " + + " }); " + + " }) " + + " }; " + + "})" + )); -* **The query**: - * To get all documents matching a specific attribute property use: - - - -{`List matchingDocuments = session - .query(Product.class, Attributes_ByName.class) - .whereEquals("Width", 10) - .toList(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the attributes list -from index 'Attributes/ByName' where Width = 10 -`} - + Map fields = new HashMap<>(); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + IndexFieldOptions allFieldsOptions = new IndexFieldOptions(); + allFieldsOptions.setStorage(FieldStorage.NO); + allFieldsOptions.setAnalyzer("StopAnalyzer"); + fields.put(Constants.Documents.Indexing.Fields.ALL_FIELDS, allFieldsOptions); + + setFields(fields); + } +} +``` + + + + +### Which analyzer is used for the query term? +The following applies when a `search()` query targets a **dynamic field**: -## CreateField syntax +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | -#### Syntax for LINQ-index: + - - -{`object CreateField(string name, object value); + + + + +#### Syntax for `AbstractIndexCreationTask` + + +```csharp +object CreateField(string name, object value); object CreateField(string name, object value, bool stored, bool analyzed); object CreateField(string name, object value, CreateFieldOptions options); -`} - +``` -#### Syntax for JavaScript-index: +#### Syntax for `AbstractJavaScriptIndexCreationTask` - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` -| Parameters | Type | Description | -|----------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **fieldName** | `string` | Name of the dynamic-index-field | -| **fieldValue** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorate.Yes` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | +| Parameters | Type | Description | +|----------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorage.Yes` | +| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | | **options** | `CreateFieldOptions` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| | **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +* The `CreateField` syntax (and the `CreateFieldOptions` shape) above describes what is available inside the server-side `map` string of an `AbstractIndexCreationTask`, + since the Java client sends the `map` source to the server for compilation. + The JavaScript `createField` function is the equivalent for `AbstractJavaScriptIndexCreationTask` map sources. + * All above examples have used the character `_` in the dynamic-index-field definition. However, using `_` is just a convention. Any other string can be used instead. * This property is Not queryable, it is only used in the index definition syntax. - The actual dynamic-index-fields that are generated are defined by the `CreateField` method. + The actual dynamic-index-fields that are generated are defined by the `CreateField` method (or the `createField` JS function). + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. - +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-nodejs.mdx b/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-nodejs.mdx index b5df2df4a7..c14184c2b5 100644 --- a/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-nodejs.mdx +++ b/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-nodejs.mdx @@ -2,533 +2,710 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. - -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. - -* In this page: - - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`createField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -## Indexing documents fields KEYS + - -#### Example - index any field under object -The following allows you to: - -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. -**The document**: - - -{`class Product \{ - constructor(id, attributes) \{ - this.id = id; - - // The KEYS under the attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - this.attributes = attributes; - \} -\} -`} - - + + +### Example - index every field under an object - - -{`// Sample document content -\{ - "attributes": \{ - "color": "Red", - "size": 42 - \} -\} -`} - - + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. + -**The index**: +* **The document**: + + + ```js + class Product { + constructor(id, attributes) { + this.id = id; + + // The KEYS under the attributes object will be dynamically indexed + // Fields added to this object after index creation time will also get indexed + this.attributes = attributes; + } + } + ``` + + + + ```json + // Sample document content + { + "attributes": { + "color": "Red", + "size": 42 + } + } + ``` + -* The following index will index any field under the `attributes` object from the document, +* **The index**: + + The following index will index any field under the `attributes` object from the document, a dynamic-index-field will be created for each such field. - New fields added to the object after index creation time will be dynamically indexed as well. - -* The actual dynamic-index-field name on which you can query will be the attribute field **key**. - e.g. Keys `color` & `size` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { - constructor() { - super(); - - const { createField } = this.mapUtils(); - - this.map("Products", p => { - return { - // Call 'createField' to generate dynamic-index-fields from the attributes object keys - // Using '_' is just a convention. Any other string can be used instead of '_' - - // The actual field name will be the key - // The actual field terms will be derived from p.attributes[key] - _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], { - indexing: "Search", - storage: false, - termVector: null - })) - }; - }); + New fields added to the object after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `color` & `size` will become the actual dynamic-index-fields. + + + ```js + class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + this.map("Products", p => { + return { + // Call 'createField' to generate dynamic-index-fields + // from the attributes object keys. + + // Using '_' is just a convention. + // Any other string can be used instead of '_' + + // The actual field name will be the key + // The actual field terms will be derived from p.attributes[key] + _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key])) + }; + }); + } } -} -`} - - - - -**The query**: - -* You can now query the generated dynamic-index fields. - Property `_` is Not queryable, it is only used in the index definition syntax. - -* To get all documents with some 'size' use: - - - - -{`const matchingDocuments = session.query({indexName: 'Products_ByAttributeKey'}) - // 'size' is a dynamic-index-field that was indexed from the attributes object - .whereEquals('size', 42) - .all(); -`} - - - - -{`// 'size' is a dynamic-index-field that was indexed from the attributes object -from index 'Products/ByAttributeKey' where size = 42 -`} - - - - - + ``` + + +* **The query**: + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _size_ use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Products/ByAttributeKey/JS" }) + // 'size' is a dynamic-index-field that was indexed from the attributes object + .whereEquals("size", 42) + .all(); + ``` + + + ```sql + // 'size' is a dynamic-index-field that was indexed from the attributes object + from index 'Products/ByAttributeKey/JS' where size = 42 + ``` + + + + + + + +### Example - index every field -#### Example - index any field -The following allows you to: - -* Define an index on a collection **without** needing any common structure between the indexed documents. -* After index is deployed, any new field added to the document will be indexed as well. - +The following example allows you to: + * Define an index on a collection **without** needing any common structure between the indexed documents. + * After the index is deployed, any new field added to any document in the collection will be indexed as well. + + - -Consider if that is a true necessity, as indexing every single field can end up costing time and disk space. - +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. -**The document**: - - -{`class Product \{ - constructor(id, firstName, lastName, title) \{ - this.id = id; - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - this.firstName = firstName; - this.lastName = lastName; - this.title = title; - // ... - \} -\} -`} - - - - - -{`// Sample document content -\{ - "firstName": "John", - "lastName": "Doe", - "title": "Engineer", - // ... -\} -`} - - -**The index**: +* **The document**: + + + ```js + class Product { + constructor(id, firstName, lastName, title) { + this.id = id; + + // All KEYS in the document will be dynamically indexed + // Fields added to the document after index creation time will also get indexed + this.firstName = firstName; + this.lastName = lastName; + this.title = title; + // ... + } + } + ``` + + + + ```json + // Sample document content + { + "firstName": "John", + "lastName": "Doe", + "title": "Engineer", + // ... + } + ``` + -* The following index will index any field from the document, +* **The index**: + + The following index will index any field from the document, a dynamic-index-field will be created for each field. - New fields added to the document after index creation time will be dynamically indexed as well. - -* The actual dynamic-index-field name on which you can query will be the field **key**. - e.g. Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + New fields added to the document after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + + + ```js + class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + // This will index EVERY FIELD under the top level of the document + this.map("Products", p => { + return { + _: Object.keys(p).map(key => createField(key, p[key])) + }; + }); + } + } + ``` + + +* **The query**: + + To get all documents with a specific _lastName_ use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Products/ByAnyField/JS" }) + // 'lastName' is a dynamic-index-field that was indexed from the document + .whereEquals("lastName", "Doe") + .all(); + ``` + + + ```sql + // 'lastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where lastName = "Doe" + ``` + + + + + + + + + + + +### Example - basic + + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + - - - -{`class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { - constructor () { - super(); +* **The document**: + + + ```js + class Product { + constructor(id, productType, pricePerUnit) { + this.id = id; + + // The VALUE of productType will be dynamically indexed + this.productType = productType; + this.pricePerUnit = pricePerUnit; + } + } + ``` + + + + ```json + // Sample document content + { + "productType": "Electronics", + "pricePerUnit": 23 + } + ``` + + +* **The index**: + + The following index will index the **value** of document field 'productType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + ```js + class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + this.map("Products", p => { + return { + // The field name will be the value of document field 'productType' + // The field terms will be derived from document field 'pricePerUnit' + _: createField(p.productType, p.pricePerUnit) + }; + }); + } + } + ``` + + +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Products/ByProductType/JS" }) + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + .whereEquals("Electronics", 23) + .all(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + from index 'Products/ByProductType/JS' where Electronics = 23 + ``` + + + + + + + +### Example - list - const { createField } = this.mapUtils(); + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. + - this.map("Products", p => { - return { - // This will index EVERY FIELD under the top level of the document - _: Object.keys(p).map(key => createField(key, p[key], { - indexing: "Search", - storage: true, - termVector: null - })) - }; - }); +* **The document**: + + + ```js + class Product { + constructor(id, name, attributes) { + this.id = id; + this.name = name; + + // For each element in this list, + // the VALUE of property 'propName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + this.attributes = attributes; + } } -} -`} - - - -**The query**: + class Attribute { + constructor(propName, propValue) { + this.propName = propName; + this.propValue = propValue; + } + } + ``` + + + + ```json + // Sample document content + { + "name": "SomeName", + "attributes": [ + { + "propName": "Color", + "propValue": "Blue" + }, + { + "propName": "Width", + "propValue": "10" + }, + { + "propName": "Length", + "propValue": "20" + }, + ... + ] + } + ``` + + +* **The index**: + + The following index will create a dynamic-index-field per item in the document's `attributes` list. + New items added to the attributes list after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the item's propName **value**. + E.g., 'propName' value `Width` will be a dynamic-index-field. + + + ```js + class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + this.map("Products", p => { + return { + // Define the dynamic-index-fields by calling 'createField' + // A dynamic-index-field will be generated for each item in 'attributes' + + // For each item, the field name will be the value of field 'propName' + // The field terms will be derived from field 'propValue' + _: p.attributes.map(item => + createField(item.propName, item.propValue)), + + // A regular index field can be defined as well: + name: p.name + }; + }); + } + } + ``` + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```js + const matchingDocuments = await session + .query({ indexName: "Attributes/ByName/JS" }) + // 'Width' is a dynamic-index-field that was indexed from the attributes list + .whereEquals("Width", 10) + .all(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the attributes list + from index 'Attributes/ByName/JS' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `indexing: "Search"` in the `createField` options. + +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. + + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. + + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. -* To get all documents with some 'lastName' use: + + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```js + class Product { + constructor(id, descriptions) { + this.id = id; + this.descriptions = descriptions; + } + } + ``` + + + + ```json + // Sample document content + { + "descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. + + Each generated dynamic field is indexed for **full-text search**. + + + ```js + class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { + super(); + + const { createField } = this.mapUtils(); + + const useSearchAnalyzerForDynamicFields = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + this.map("Products", product => { + return { + _: Object.keys(product.descriptions).map(key => + createField( + "Description_" + key, + product.descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: "Search", + storage: false + })) + }; + }); + + this.storeAllFields("No"); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + this.configuration[useSearchAnalyzerForDynamicFields] = "true"; + } + } + ``` + - - - -{`const matchingDocuments = session.query({ indexName: 'Products_ByAnyField_JS' }) - // 'lastName' is a dynamic-index-field that was indexed from the document - .whereEquals('lastName', 'Doe') - .all(); -`} - - - - -{`// 'lastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where lastName = "Doe" -`} - - - +* **Full-text search query**: - + Query the generated dynamic field by its field name. + In this example, the query targets the generated field `Description_English`. + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). -## Indexing documents fields VALUES + + + ```js + const results = await session + .query({ indexName: "Products/ByLocalizedDescription/JS" }) + .search("Description_English", "north wind") + .all(); + ``` + - -#### Example - basic -This example shows: - -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. -* For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---index-a-list-of-properties) below. -**The document**: - - -{`class Product \{ - constructor(id, productType, pricePerUnit) \{ - this.id = id; - - // The VALUE of productType will be dynamically indexed - this.productType = productType; - this.pricePerUnit = pricePerUnit; - \} -\} -`} - - + + ```sql + from index "Products/ByLocalizedDescription/JS" + where search(Description_English, "north wind") + ``` + + - - -{`// Sample document content -\{ - "productType": "Electronics", - "pricePerUnit": 23 -\} -`} - - +--- -**The index**: + + +### Configuring analyzers for dynamic fields -* The following index will index the **value** of document field 'productType'. +The `createField` options object does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. -* This value will be the dynamic-index-field name on which you can query. - e.g. Field value `Electronics` will be the dynamic-index-field. +--- - - - -{`class Products_ByProductType extends AbstractCsharpIndexCreationTask { - constructor () { - super(); +#### Configure analyzer for a specific dynamic field - // The field name will be the value of document field 'productType' - // The field terms will be derived from document field 'pricePerUnit' - this.map = "docs.Products.Select(p => new { " + - " _ = this.CreateField(p.productType, p.pricePerUnit) " + - "})"; - } -} -`} - - - - -{`class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask { - constructor () { + +```js +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { super(); const { createField } = this.mapUtils(); - this.map("Products", p => { + this.map("Products", product => { return { - _: [ - // The field name will be the value of document field 'productType' - // The field terms will be derived from document field 'pricePerUnit' - createField(p.productType, p.pricePerUnit, { - indexing: "Search", - storage: false, - termVector: null - }) - ] + _: Object.keys(product.descriptions).map(key => + createField( + "Description_" + key, + product.descriptions[key], + { + indexing: "Search", + storage: false + })) }; }); - } -} -`} - - - - -**The query**: - -* To get all documents of some product type having a specific price per unit use: - - - - -{`const matchingDocuments = session.query({ indexName: 'Products_ByProductType' }) - // 'Electronics' is the dynamic-index-field that was indexed from document field 'productType' - .whereEquals('Electronics', 23) - .all(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'productType' -from index 'Products/ByProductType' where Electronics = 23 -`} - - - - - - -#### Example - list -The following allows you to: - -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. -**The document**: - - -{`class Product \{ - constructor(id, name, attributes) \{ - this.id = id; - this.name = name; - - // For each element in this list, the VALUE of property 'propName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - this.attributes = attributes; - \} -\} - -class Attribute \{ - constructor(propName, propValue) \{ - this.propName = propName; - this.propValue = propValue; - \} -\} -`} - - - - - -{`// Sample document content -\{ - "name": "SomeName", - "attributes": [ - \{ - "propName": "Color", - "propValue": "Blue" - \}, - \{ - "propName": "Width", - "propValue": "10" - \}, - \{ - "propName": "Length", - "propValue": "20" - \}, - ... - ] -\} -`} - - - -**The index**: - -* The following index will create a dynamic-index-field per item in the document's `attributes` list. - New items added to the attributes list after index creation time will be dynamically indexed as well. - -* The actual dynamic-index-field name on which you can query will be the item's propName **value**. - e.g. 'propName' value `Width` will be a dynamic-index-field. - - - - -{`class Attributes_ByName extends AbstractCsharpIndexCreationTask -{ - constructor () { - super(); + this.storeAllFields("No"); - // For each attribute item, the field name will be the value of field 'propName' - // The field terms will be derived from field 'propValue' - // A regular-index-field (Name) is defined as well - this.map = - "docs.Products.Select(p => new { " + - " _ = p.attributes.Select(item => this.CreateField(item.propName, item.propValue)), " + - " Name = p.name " + - "})"; + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + this.analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -{`class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask { - constructor () { + +#### Configure a fallback analyzer for all fields + + +```js +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { + constructor() { super(); const { createField } = this.mapUtils(); - this.map("Products", p => { + this.map("Products", product => { return { - // For each item, the field name will be the value of field 'propName' - // The field terms will be derived from field 'propValue' - _: p.attributes.map(item => createField(item.propName, item.propValue, { - indexing: "Search", - storage: true, - termVector: null - })), - - // A regular-index-field can be defined as well: - Name: p.name + _: Object.keys(product.descriptions).map(key => + createField( + "Description_" + key, + product.descriptions[key], + { + indexing: "Search", + storage: false + })) }; }); - } -} -`} - - - - -**The query**: -* To get all documents matching a specific attribute property use: + this.storeAllFields("No"); - - - -{`const matchingDocuments = session.query({ indexName: 'Attributes/ByName' }) - // 'Width' is a dynamic-index-field that was indexed from the attributes list - .whereEquals('Width', 10) - .all(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the attributes list -from index 'Attributes/ByName' where Width = 10 -`} - + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + this.analyze(CONSTANTS.Documents.Indexing.Fields.ALL_FIELDS, "StopAnalyzer"); + } +} +``` - - - + + + + +### Which analyzer is used for the query term? -## CreateField syntax +The following applies when a `search()` query targets a **dynamic field**: -#### Syntax for LINQ-index: +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `this.analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `this.analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | - - -{`object CreateField(string name, object value); + -object CreateField(string name, object value, bool stored, bool analyzed); + -object CreateField(string name, object value, CreateFieldOptions options); -`} - - + -#### Syntax for JavaScript-index: +#### Syntax: - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` -| Parameters | Type | Description | -|----------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **fieldName** | `string` | Name of the dynamic-index-field | -| **fieldValue** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorate.Yes` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | -| **options** | `CreateFieldOptions` | Dynamic-index-field options | +| Parameters | Type | Description | +|----------------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **options** | `object` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| -| **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| options object | | | +|-----------------|--------------------|-----------------------------------------------------------------------------------| +| **storage** | `boolean` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | +| **indexing** | `FieldIndexing` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **termVector** | `FieldTermVector` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -536,21 +713,19 @@ object CreateField(string name, object value, CreateFieldOptions options); However, using `_` is just a convention. Any other string can be used instead. * This property is Not queryable, it is only used in the index definition syntax. - The actual dynamic-index-fields that are generated are defined by the `CreateField` method. + The actual dynamic-index-fields that are generated are defined by the `createField` method. + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. - +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-php.mdx b/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-php.mdx index 09c58cc9fc..6992310715 100644 --- a/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-php.mdx +++ b/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-php.mdx @@ -2,537 +2,969 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. + -* In this page: + - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) + + +### Example - index every field under an object + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. -## Indexing documents fields KEYS - -## Example - index any field under object +* **The document**: + + + ```php + use Ds\Map as DSMap; - + class Product + { + public ?string $id = null; -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + // The KEYS under the attributes object will be dynamically indexed. + // Fields added to this object after index creation time will also get indexed. + public ?DSMap $attributes = null; - - -* **The document**: - - -{`use Ds\\Map as DSMap; - -class Product -\{ - private ?string $id = null; - - // The KEYS under the Attributes object will be dynamically indexed - // Fields added to this object after index creation time will also get indexed - public ?DSMap $attributes = null; -\} -`} - - - - -{`// Sample document content -\{ - "Attributes": \{ - "Color": "Red", - "Size": 42 - \} -\} -`} - - + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "attributes": { + "Color": "Red", + "Size": 42 + } + } + ``` + * **The index**: - The below index will index any field under the `Attributes` object from the document, + + The following index will index any field under the `attributes` object from the document, a dynamic-index-field will be created for each such field. New fields added to the object after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. - - - -{`class Products_ByAttributeKey extends AbstractIndexCreationTask -{ - public function __construct() + + + ```php + class Products_ByAttributeKey extends AbstractIndexCreationTask { - parent::__construct(); - - $this->map = "from p in docs.Products select new {" . - "_ = p.attributes.Select(item => CreateField(item.Key, item.Value))" . - "}"; + public function __construct() + { + parent::__construct(); + + // Call 'CreateField' to generate dynamic-index-fields + // from the attributes object keys. + // + // Using '_' is just a convention. + // Any other string can be used instead of '_'. + // + // The actual field name will be 'item.Key'. + // The actual field terms will be derived from 'item.Value'. + $this->map = + "from p in docs.Products " . + "select new { " . + " _ = p.attributes.Select(item => CreateField(item.Key, item.Value)) " . + "}"; + } } -} -`} - - - - -{`class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask -{ - public function __construct() + ``` + + + ```php + class Products_ByAttributeKey_JS extends AbstractJavaScriptIndexCreationTask { - parent::__construct(); - - $this->setMaps([ - "map('Products', function (p) { " . - " return { " . - " _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], " . - " { indexing: 'Search', storage: false, termVector: null })) " . - " }; " . - "}) " - ]); + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (p) { + return { + // The field name will be the key + // The field terms will be derived from the corresponding value + _: Object.keys(p.attributes) + .map(key => createField(key, p.attributes[key], null)) + }; + })" + ]); + } } -} -`} - - - - + ``` + + * **The query**: - * You can now query the generated dynamic-index fields. - * To get all documents with some 'size' use: - - - -{`$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Products_ByAttributeKey::class) - // 'Size' is a dynamic-index-field that was indexed from the Attributes object - ->whereEquals("Size", 42) - ->toList(); -`} - - - - -{`// 'Size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey' where Size = 42 -`} - - - - -## Example - index any field - - + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _Size_ use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Products_ByAttributeKey::class) + // 'Size' is a dynamic-index-field that was indexed from the attributes object + ->whereEquals("Size", 42) + ->toList(); + ``` + + + ```sql + // 'Size' is a dynamic-index-field that was indexed from the attributes object + from index 'Products/ByAttributeKey' where Size = 42 + ``` + + + + + Dynamic index fields are queried with _documentQuery_ or _RQL_ because the field names (e.g. `Size`, `Color`) + are generated at indexing time and are not properties on the _Product_ class. + + Use the string-based `whereEquals("Size", 42)` or RQL `where Size = 42` instead. + + + + + + +### Example - index every field + +The following example allows you to: * Define an index on a collection **without** needing any common structure between the indexed documents. - * After index is deployed, any new field added to the document will be indexed as well. - + * After the index is deployed, any new field added to any document in the collection will be indexed as well. - + -Consider whether this is really necessary, as indexing every single field can end up costing time and disk space. +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`class Product -\{ - private ?string $id = null; - - // All KEYS in the document will be dynamically indexed - // Fields added to the document after index creation time will also get indexed - public ?string $firstName = null; - public ?string $lastName = null; - public ?string $title = null; - - // ... getters and setters -\} -`} - - + + + ```php + class Product + { + public ?string $id = null; - - -{`// Sample document content - \{ - "FirstName": "John", - "LastName": "Doe", - "Title": "Engineer", + // All KEYS in the document will be dynamically indexed. + // Fields added to the document after index creation time will also get indexed. + public ?string $firstName = null; + public ?string $lastName = null; + public ?string $title = null; // ... -\} -`} - - + + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "firstName": "John", + "lastName": "Doe", + "title": "Engineer" + // ... + } + ``` + * **The index**: - The below index will index any field from the document, + + The following index will index any field from the document, a dynamic-index-field will be created for each field. New fields added to the document after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask -{ - public function __construct() + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `firstName` & `lastName` will become the actual dynamic-index-fields. + + The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection. + + + + ```php + class Products_ByAnyField_JS extends AbstractJavaScriptIndexCreationTask { - parent::__construct(); - - // This will index EVERY FIELD under the top level of the document - $this->setMaps([ - "map('Products', function (p) { - return { - _: Object.keys(p).map(key => createField(key, p[key], - { indexing: 'Search', storage: true, termVector: null })) - } - })" - ]); + public function __construct() + { + parent::__construct(); + + // This will index EVERY FIELD under the top level of the document + $this->setMaps([ + "map('Products', function (p) { + return { + _: Object.keys(p).map(key => createField(key, p[key], null)) + }; + })" + ]); + } } -} -`} - - - + ``` + + * **The query**: - * To get all documents with some 'LastName' use: - - - -{`$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Products_ByAnyField_JS::class) - // 'LastName' is a dynamic-index-field that was indexed from the document - ->whereEquals("LastName", "Doe") - ->toList(); -`} - - - - -{`// 'LastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where LastName = "Doe" -`} - - - + + To get all documents with a specific _lastName_ use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Products_ByAnyField_JS::class) + // 'lastName' is a dynamic-index-field that was indexed from the document + ->whereEquals("lastName", "Doe") + ->toList(); + ``` + + + ```sql + // 'lastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where lastName = "Doe" + ``` + + + + + + + + + + + +### Example - basic + + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + +* **The document**: + + + ```php + class Product + { + public ?string $id = null; + // The VALUE of productType will be dynamically indexed + public ?string $productType = null; + public ?int $pricePerUnit = null; -## Indexing documents fields VALUES + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "productType": "Electronics", + "pricePerUnit": 23 + } + ``` + -## Example - basic +* **The index**: + + The following index will index the **value** of document field 'productType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. - + + + ```php + class Products_ByProductType extends AbstractIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + // Call 'CreateField' to generate the dynamic-index-fields. + // The field name will be the value of document field 'productType'. + // The field terms will be derived from document field 'pricePerUnit'. + $this->map = + "from p in docs.Products " . + "select new { " . + " _ = CreateField(p.productType, p.pricePerUnit) " . + "}"; + } + } + ``` + + + ```php + class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (p) { + return { + // The field name will be the value of document field 'productType' + // The field terms will be derived from document field 'pricePerUnit' + _: createField(p.productType, p.pricePerUnit, null) + }; + })" + ]); + } + } + ``` + + -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Products_ByProductType::class) + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + ->whereEquals("Electronics", 23) + ->toList(); + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'productType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + + +### Example - list + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. * **The document**: - - -{`class Product -\{ - public ?string $id = null; - - // The VALUE of ProductType will be dynamically indexed - public ?string $productType = null; - public ?int $pricePerUnit = null; - - // ... getters and setters -\} -`} - - + + + ```php + class Product + { + public ?string $id = null; + public ?string $name = null; - - -{`// Sample document content -\{ - "ProductType": "Electronics", - "PricePerUnit": 23 -\} -`} - - + // For each element in this list, + // the VALUE of property 'propName' will be dynamically indexed. + // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + public ?AttributeList $attributes = null; + + // ... getters and setters + } + + class Attribute + { + public ?string $propName = null; + public ?string $propValue = null; + + // ... getters and setters + } + + class AttributeList extends TypedList + { + protected function __construct() + { + parent::__construct(Attribute::class); + } + } + ``` + + + + ```json + // Sample document content + { + "name": "SomeName", + "attributes": [ + { + "propName": "Color", + "propValue": "Blue" + }, + { + "propName": "Width", + "propValue": "10" + }, + { + "propName": "Length", + "propValue": "20" + } + // ... + ] + } + ``` + * **The index**: - The below index will index the **value** of document field 'ProductType'. + + The following index will create a dynamic-index-field per item in the document's `attributes` list. + New items added to the attributes list after index creation time will be dynamically indexed as well. - This value will be the dynamic-index-field name on which you can query. - E.g., Field value `Electronics` will be the dynamic-index-field. + The actual dynamic-index-field name on which you can query will be the item's propName **value**. + E.g., 'propName' value `Width` will be a dynamic-index-field. + + + + ```php + class Attributes_ByName extends AbstractIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + // Define the dynamic-index-fields by calling 'CreateField'. + // A dynamic-index-field will be generated for each item in 'attributes'. + // + // For each item, the field name will be the value of field 'propName'. + // The field terms will be derived from field 'propValue'. + $this->map = + "from p in docs.Products " . + "select new { " . + " _ = p.attributes.Select(item => CreateField(item.propName, item.propValue)), " . + " Name = p.name " . + "}"; + } + } + ``` + + + ```php + class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask + { + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (p) { + return { + // For each item, + // the field name will be the value of field 'propName' + // The field terms will be derived from field 'propValue' + _: p.attributes.map(item => createField( + item.propName, item.propValue, null)), + + // A regular index field can be defined as well: + Name: p.name + }; + })" + ]); + } + } + ``` + + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```php + /** @var array $matchingDocuments */ + $matchingDocuments = $session + ->advanced() + ->documentQuery(Product::class, Attributes_ByName::class) + // 'Width' is a dynamic-index-field that was indexed from the attributes list + ->whereEquals("Width", 10) + ->toList(); + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `FieldIndexing::search()` in `CreateFieldOptions`. + +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. + + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. + + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. + + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + + + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```php + use Ds\Map as DSMap; + + class Product + { + public ?string $id = null; + public ?DSMap $descriptions = null; + + // ... getters and setters + } + ``` + + + + ```json + // Sample document content + { + "descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + + +* **The index**: + + The following index creates one dynamic field for each entry in the `descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. - + Each generated dynamic field is indexed for **full-text search**. + + + + ```php + class Products_ByLocalizedDescription extends AbstractIndexCreationTask + { + private const USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public function __construct() + { + parent::__construct(); + + $this->map = + "from product in docs.Products " . + "select new { " . + " _ = product.descriptions.Select(x => " . + " CreateField(" . + " \"Description_\" + x.Key, " . + " x.Value, " . + " new CreateFieldOptions { " . + // Index each generated dynamic field for FTS + " Indexing = FieldIndexing.Search, " . + " Storage = FieldStorage.No " . + " })) " . + "}"; + + $this->storeAllFields(FieldStorage::no()); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + $this->getConfiguration()[self::USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS] = "true"; + } + } + ``` + + + ```php + class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask + { + private const USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery"; + + public function __construct() + { + parent::__construct(); + + $this->setMaps([ + "map('Products', function (product) { + return { + _: Object.keys(product.descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: 'Search', + storage: 'No' + }); + }) + }; + })" + ]); + + // Apply a storage default to every generated field + $fields = new IndexFieldOptionsArray(); + $allFieldsOptions = new IndexFieldOptions(); + $allFieldsOptions->setStorage(FieldStorage::no()); + $fields->offsetSet(DocumentsIndexingFields::ALL_FIELDS, $allFieldsOptions); + $this->setFields($fields); + + // Set this configuration key to true so that search() queries on dynamic fields + // will use the 'Default Search Analyzer' when the generated field name + // has no explicit analyzer configuration. + $this->getConfiguration()[self::USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS] = "true"; + } + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```php + /** @var array $results */ + $results = $session->advanced() + ->documentQuery(Product::class, Products_ByLocalizedDescription::class) + ->search("Description_English", "north wind") + ->toList(); + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + + +--- + + + +### Configuring analyzers for dynamic fields + +`CreateFieldOptions` does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field + + - -{`class Products_ByProductType extends AbstractIndexCreationTask +```php +class Products_ByLocalizedDescription extends AbstractIndexCreationTask { public function __construct() { parent::__construct(); - // Call 'CreateField' to generate the dynamic-index-fields - // The field name will be the value of document field 'ProductType' - // The field terms will be derived from document field 'PricePerUnit' - $this->map = "docs.Products.Select(p => new { " . - " _ = this.CreateField(p.productType, p.pricePerUnit) " . - "})"; + $this->map = + "from product in docs.Products " . + "select new { " . + " _ = product.descriptions.Select(x => " . + " CreateField(" . + " \"Description_\" + x.Key, " . + " x.Value, " . + " new CreateFieldOptions { " . + " Indexing = FieldIndexing.Search, " . + " Storage = FieldStorage.No " . + " })) " . + "}"; + + $this->storeAllFields(FieldStorage::no()); + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + $this->analyze("Description_English", "StopAnalyzer"); } } -`} - +``` - - -{`class Products_ByProductType_JS extends AbstractJavaScriptIndexCreationTask + +```php +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { public function __construct() { parent::__construct(); $this->setMaps([ - "map('Products', function (p) { + "map('Products', function (product) { return { - _: createField(p.ProductType, p.PricePerUnit, - { indexing: 'Search', storage: true, termVector: null }) + _: Object.keys(product.descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" ]); - } -} -`} - - - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Products_ByProductType::class) -// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -->whereEquals("Electronics", 23) -->toList(); -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -from index 'Products/ByProductType' where Electronics = 23 -`} - - - - -## Example - list - - -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. + // Apply a storage default to every generated field + + // Configure an analyzer for a known generated dynamic field name + // ============================================================== + $fields = new IndexFieldOptionsArray(); - + $allFieldsOptions = new IndexFieldOptions(); + $allFieldsOptions->setStorage(FieldStorage::no()); + $fields->offsetSet(DocumentsIndexingFields::ALL_FIELDS, $allFieldsOptions); -* **The document**: - - -{`class Product -\{ - public ?string $id = null; - public ?string $name = null; - - // For each element in this list, the VALUE of property 'PropName' will be dynamically indexed - // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields - public ?AttributeList $attributes = null; - - // ... getters and setters -\} - -class Attribute -\{ - public ?string $propName = null; - public ?string $propValue = null; - - // ... getters and setters -\} - -class AttributeList extends TypedList -\{ - protected function __construct() - \{ - parent::__construct(Attribute::class); - \} -\} -`} - - + $englishFieldOptions = new IndexFieldOptions(); + $englishFieldOptions->setAnalyzer("StopAnalyzer"); + $fields->offsetSet("Description_English", $englishFieldOptions); - - -{`// Sample document content -\{ -Name": "SomeName", -Attributes": [ - \{ - "PropName": "Color", - "PropValue": "Blue" - \}, - \{ - "PropName": "Width", - "PropValue": "10" - \}, - \{ - "PropName": "Length", - "PropValue": "20" - \}, - ... - -\} -`} - + $this->setFields($fields); + } +} +``` + -* **The index**: - The below index will create a dynamic-index-field per item in the document's `Attributes` list. - New items added to the Attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's PropName **value**. - E.g., 'PropName' value `width` will be a dynamic-index-field. +#### Configure a fallback analyzer for all fields - + - -{`class Attributes_ByName extends AbstractIndexCreationTask +```php +class Products_ByLocalizedDescription extends AbstractIndexCreationTask { public function __construct() { parent::__construct(); - // Define the dynamic-index-fields by calling CreateField - // A dynamic-index-field will be generated for each item in the Attributes list + $this->map = + "from product in docs.Products " . + "select new { " . + " _ = product.descriptions.Select(x => " . + " CreateField(" . + " \"Description_\" + x.Key, " . + " x.Value, " . + " new CreateFieldOptions { " . + " Indexing = FieldIndexing.Search, " . + " Storage = FieldStorage.No " . + " })) " . + "}"; - // For each item, the field name will be the value of field 'PropName' - // The field terms will be derived from field 'PropValue' + $this->storeAllFields(FieldStorage::no()); - $this->map = - "docs.Products.Select(p => new { " . - " _ = p.attributes.Select(item => this.CreateField(item.propName, item.propValue)), " . - " Name = p.name " . - "})"; + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + $this->analyze(DocumentsIndexingFields::ALL_FIELDS, "StopAnalyzer"); } } -`} - +``` - - -{`class Attributes_ByName_JS extends AbstractJavaScriptIndexCreationTask + +```php +class Products_ByLocalizedDescription_JS extends AbstractJavaScriptIndexCreationTask { public function __construct() { parent::__construct(); $this->setMaps([ - "map('Products', function (p) { + "map('Products', function (product) { return { - _: p.Attributes.map(item => createField(item.PropName, item.PropValue, - { indexing: 'Search', storage: true, termVector: null })), - Name: p.Name + _: Object.keys(product.descriptions) + .map(function (key) { + return createField( + 'Description_' + key, + product.descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; })" ]); + + // Use this as a fallback for fields that do not have + // their own explicit analyzer configuration + // ================================================== + $fields = new IndexFieldOptionsArray(); + $allFieldsOptions = new IndexFieldOptions(); + $allFieldsOptions->setStorage(FieldStorage::no()); + $allFieldsOptions->setAnalyzer("StopAnalyzer"); + $fields->offsetSet(DocumentsIndexingFields::ALL_FIELDS, $allFieldsOptions); + $this->setFields($fields); } } -`} - - - - -* **The query**: - To get all documents matching a specific attribute property use: - - - -{`/** @var array $matchingDocuments */ -$matchingDocuments = $session - ->advanced() - ->documentQuery(Product::class, Attributes_ByName::class) - // 'Width' is a dynamic-index-field that was indexed from the Attributes list - ->whereEquals("Width", 10) - ->toList(); -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the Attributes list -from index 'Attributes/ByName' where Width = 10 -`} - +``` + + + + +### Which analyzer is used for the query term? +The following applies when a `search()` query targets a **dynamic field**: -## CreateField syntax +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `$this->analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `$this->analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | - - -{`object CreateField(string name, object value); + + + + + + +#### Syntax for LINQ-index: + + +```php +object CreateField(string name, object value); object CreateField(string name, object value, bool stored, bool analyzed); object CreateField(string name, object value, CreateFieldOptions options); -`} - +``` -| Parameters | Type | Description | -|------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **name** | `string` | Name of the dynamic-index-field | -| **value** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`False` - will set `FieldStorage.NO` (default value)
`True` - will set `FieldStorate.YES` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`None` - `FieldIndexing.Default` (default value)
`False` - `FieldIndexing.Exact`
`True` - `FieldIndexing.Search` | -| **options** | `CreateFieldOptions` | Dynamic-index-field options | +#### Syntax for JavaScript-index: -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| -| **Storage** | `FieldStorage` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing` | | -| **TermVector** | `FieldTermVector` | | + +```js +createField(fieldName, fieldValue, options); // returns object +``` + + +| Parameters | Type | Description | +|----------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage::no()` (default value)
`true` - will set `FieldStorage::yes()` | +| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing::default()` (default value)
`false` - `FieldIndexing::exact()`
`true` - `FieldIndexing::search()` | +| **options** | `CreateFieldOptions` | Dynamic-index-field options | + +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| +| **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -544,17 +976,15 @@ object CreateField(string name, object value, CreateFieldOptions options); + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. - +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). + ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) - + ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-python.mdx b/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-python.mdx index 69514a29e2..6862d18067 100644 --- a/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-python.mdx +++ b/versioned_docs/version-7.1/indexes/content/_using-dynamic-fields-python.mdx @@ -1,490 +1,886 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; -* In RavenDB different documents can have different shapes. - Documents are schemaless - new fields can be added or removed as needed. - -* For such dynamic data, you can define indexes with **dynamic-index-fields**. - -* This allows querying the index on fields that aren't yet known at index creation time, - which is very useful when working on highly dynamic systems. - -* Any value type can be indexed, string, number, date, etc. - -* An index definition can contain both dynamic-index-fields and regular-index-fields. - -* In this page: - - * [Indexing documents fields KEYS](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-keys) - * [Example - index any field under object](../../indexes/using-dynamic-fields.mdx#example---index-any-field-under-object) - * [Example - index any field](../../indexes/using-dynamic-fields.mdx#example---index-any-field) - * [Indexing documents fields VALUES](../../indexes/using-dynamic-fields.mdx#indexing-documents-fields-values) - * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) - * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) - * [CreateField syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) - * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields--terms-view) +* In RavenDB, different documents can have different shapes. + Documents are schemaless - new fields can be added or removed as needed. + For such dynamic data, you can define indexes with **dynamic-index-fields**. + +* Dynamic-index-fields are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time, + which is very useful when working with highly dynamic systems. + +* Any value type can be indexed, string, number, date, etc. + An index definition can contain both dynamic-index-fields and regular-index-fields. + +* In this article: + * [Indexing field KEYS](../../indexes/using-dynamic-fields.mdx#indexing-field-keys) + * [Example - index every field under an object](../../indexes/using-dynamic-fields.mdx#example---index-every-field-under-an-object) + * [Example - index every field](../../indexes/using-dynamic-fields.mdx#example---index-every-field) + * [Indexing field VALUES](../../indexes/using-dynamic-fields.mdx#indexing-field-values) + * [Example - basic](../../indexes/using-dynamic-fields.mdx#example---basic) + * [Example - list](../../indexes/using-dynamic-fields.mdx#example---list) + * [Indexing dynamic fields for full-text search](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + * [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) + * [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + * [`CreateField` syntax](../../indexes/using-dynamic-fields.mdx#createfield-syntax) + * [Indexed fields & terms view](../../indexes/using-dynamic-fields.mdx#indexed-fields-terms-view) -## Indexing documents fields KEYS - -## Example - index any field under object + - - -* Index any field that is under the some object from the document. -* After index is deployed, any new field added to the this object will be indexed as well. + + +### Example - index every field under an object + +The following example allows you to: + * Index any field that is under the same object from the document. + * After index is deployed, any new field added to this object will be indexed as well. * **The document**: - - -{`class Product: - def __init__(self, Id: str = None, attributes: Dict[str, object] = None): - self.Id = Id - - # The KEYS under the Attributes object will be dynamically indexed - # Fields added to this object after index creation time will also get indexed - self.attributes = attributes -`} - - - - -{`// Sample document content -\{ - "Attributes": \{ - "Color": "Red", - "Size": 42 - \} -\} -`} - - + + + ```python + class Product: + def __init__(self, Id: str = None, Attributes: Dict[str, object] = None): + self.Id = Id + + # The KEYS under the Attributes object will be dynamically indexed + # Fields added to this object after index creation time will also get indexed + self.Attributes = Attributes + ``` + + + + ```json + // Sample document content + { + "Attributes": { + "Color": "Red", + "Size": 42 + } + } + ``` + * **The index**: - The below index will index any field under the `Attributes` object from the document, + + The following index will index any field under the `Attributes` object from the document, a dynamic-index-field will be created for each such field. New fields added to the object after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the attribute field **key**. - E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAttributeKey(AbstractIndexCreationTask): - def __init__(self): - super().__init__() - self.map = ( - "from p in docs.Products select new {" - "_ = p.attributes.Select(item => CreateField(item.Key, item.Value))" - "}" - ) -`} - - - - -{`class Products_ByAttributeKey_JS(AbstractJavaScriptIndexCreationTask): - def __init__(self): - super().__init__() - self.maps = { - """ - map('Products', function (p) { - return { - _: Object.keys(p.attributes).map(key => createField(key, p.attributes[key], - { indexing: 'Search', storage: true, termVector: null })) - }; - }) - """ - } -`} - - - - + The actual dynamic-index-field name on which you can query will be the attribute field **key**. + E.g., Keys `Color` & `Size` will become the actual dynamic-index-fields. + + + + ```python + class Products_ByAttributeKey(AbstractIndexCreationTask): + def __init__(self): + super().__init__() + self.map = ( + "from p in docs.Products select new { " + # Call 'CreateField' to generate dynamic-index-fields + # from the Attributes object keys. + + # Using '_' is just a convention. + # Any other string can be used instead of '_' + + # The actual field name will be 'item.Key' + # The actual field terms will be derived from 'item.Value' + "_ = p.Attributes.Select(item => CreateField(item.Key, item.Value)) " + "}" + ) + ``` + + + ```python + class Products_ByAttributeKey_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (p) { + return { + // The field name will be the key + // The field terms will be derived from the corresponding value + _: Object.keys(p.Attributes) + .map(key => createField(key, p.Attributes[key])) + }; + }) + """ + ] + ``` + + * **The query**: - * You can now query the generated dynamic-index fields. - * To get all documents with some 'size' use: - - - -{`matching_documents = list( - session.query_index_type(Products_ByAttributeKey, Product) - # 'size' is a dynamic-index-field that was indexed from the attributes object - .where_equals("size", 42) -) -`} - - - - -{`// 'Size' is a dynamic-index-field that was indexed from the Attributes object -from index 'Products/ByAttributeKey' where Size = 42 -`} - - - - -## Example - index any field - - + + You can now query the generated dynamic-index fields. + The `_` property is Not queryable but used only in the index definition syntax. + + To get all documents with a specific _Size_ use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Products_ByAttributeKey, Product) + # 'Size' is a dynamic-index-field that was indexed from the Attributes object + .where_equals("Size", 42) + ) + ``` + + + ```sql + // 'Size' is a dynamic-index-field that was indexed from the Attributes object + from index 'Products/ByAttributeKey' where Size = 42 + ``` + + + + + Dynamic index fields are queried with _document_query_ or _RQL_ because the field names (e.g. `Size`, `Color`) + are generated at indexing time and are not properties on the _Product_ class. + + Use the string-based `where_equals("Size", 42)` or RQL `where Size = 42`. + + + + + + +### Example - index every field + +The following example allows you to: * Define an index on a collection **without** needing any common structure between the indexed documents. - * After index is deployed, any new field added to the document will be indexed as well. - + * After the index is deployed, any new field added to any document in the collection will be indexed as well. - + -Consider whether this is really necessary, as indexing every single field can end up costing time and disk space. +* Use this approach with care. +* Indexing every field can increase indexing time, memory usage, and disk space, + especially for large or frequently updated documents. * **The document**: - - -{`class Product: - def __init__(self, Id: str = None, first_name: str = None, last_name: str = None, title: str = None): - self.Id = Id - - # All KEYS in the document will be dynamically indexes - # Fields added to the document after index creation time wil also get indexed - self.first_name = first_name - self.last_name = last_name - self.title = title - # ... -`} - - - - - -{`// Sample document content - \{ + + + ```python + class Product: + def __init__( + self, + Id: str = None, + FirstName: str = None, + LastName: str = None, + Title: str = None, + ): + self.Id = Id + + # All KEYS in the document will be dynamically indexed + # Fields added to the document after index creation time will also get indexed + self.FirstName = FirstName + self.LastName = LastName + self.Title = Title + # ... + ``` + + + + ```json + // Sample document content + { "FirstName": "John", "LastName": "Doe", "Title": "Engineer", // ... -\} -`} - - + } + ``` + * **The index**: - The below index will index any field from the document, + + The following index will index any field from the document, a dynamic-index-field will be created for each field. New fields added to the document after index creation time will be dynamically indexed as well. - The actual dynamic-index-field name on which you can query will be the field **key**. - E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. - - - - -{`class Products_ByAnyField_JS(AbstractJavaScriptIndexCreationTask): - def __init__(self): - super().__init__() - # This will index EVERY FIELD under the top level of the document - self.maps = { - """ - map('Products', function (p) { - return { - _: Object.keys(p).map(key => createField(key, p[key], - { indexing: 'Search', storage: true, termVector: null })) - } - }) - """ - } -`} - - - + The actual dynamic-index-field name on which you can query will be the field **key**. + E.g., Keys `FirstName` & `LastName` will become the actual dynamic-index-fields. + + The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with `Object.keys(...)`. + LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection. + + + + ```python + class Products_ByAnyField_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + # This will index EVERY FIELD under the top level of the document + self.maps = [ + """ + map('Products', function (p) { + return { + _: Object.keys(p).map(key => createField(key, p[key])) + }; + }) + """ + ] + ``` + + * **The query**: - * To get all documents with some 'LastName' use: - - - -{`# 'last_name' is a dynamic-index-field that was indexed from the document -matching_documents = list( - session.query_index_type(Products_ByAnyField_JS, Product).where_equals("last_name", "Doe") -) -`} - - - - -{`// 'LastName' is a dynamic-index-field that was indexed from the document -from index 'Products/ByAnyField/JS' where LastName = "Doe" -`} - - - + + To get all documents with a specific _LastName_ use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Products_ByAnyField_JS, Product) + # 'LastName' is a dynamic-index-field that was indexed from the document + .where_equals("LastName", "Doe") + ) + ``` + + + ```sql + // 'LastName' is a dynamic-index-field that was indexed from the document + from index 'Products/ByAnyField/JS' where LastName = "Doe" + ``` + + + + + + + + + + + +### Example - basic + + +The following example shows: + * Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. + * Documents can then be queried based on those indexed values. + * For a more practical usage see the [Example](../../indexes/using-dynamic-fields.mdx#example---list) below. + + +* **The document**: + + + ```python + class Product: + def __init__(self, Id: str = None, ProductType: str = None, PricePerUnit: int = None): + self.Id = Id + + # The VALUE of ProductType will be dynamically indexed + self.ProductType = ProductType + self.PricePerUnit = PricePerUnit + ``` + + + + ```json + // Sample document content + { + "ProductType": "Electronics", + "PricePerUnit": 23 + } + ``` + + +* **The index**: + + The following index will index the **value** of document field 'ProductType'. + + This value will be the dynamic-index-field name on which you can query. + E.g., Field value `Electronics` will be the dynamic-index-field. + + + + ```python + class Products_ByProductType(AbstractIndexCreationTask): + def __init__(self): + super().__init__() + self.map = ( + "from p in docs.Products select new { " + # Call 'CreateField' to generate the dynamic-index-fields + # The field name will be the value of document field 'ProductType' + # The field terms will be derived from document field 'PricePerUnit' + "_ = CreateField(p.ProductType, p.PricePerUnit) " + "}" + ) + ``` + + + ```python + class Products_ByProductType_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (p) { + return { + // The field name will be the value of document field 'ProductType' + // The field terms will be derived from document field 'PricePerUnit' + _: createField(p.ProductType, p.PricePerUnit) + }; + }) + """ + ] + ``` + + + +* **The query**: + + To get all documents of some product type having a specific price per unit use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Products_ByProductType, Product) + # 'Electronics' is the dynamic-index-field that was indexed + # from document field 'ProductType' + .where_equals("Electronics", 23) + ) + ``` + + + ```sql + // 'Electronics' is the dynamic-index-field that was indexed + // from document field 'ProductType' + from index 'Products/ByProductType' where Electronics = 23 + ``` + + + + + + + +### Example - list + + +The following example allows you to: + * Index **values** from items in a list + * After index is deployed, any item added to this list in the document will be dynamically indexed as well. + + +* **The document**: + + + ```python + class Attribute: + def __init__(self, PropName: str = None, PropValue: str = None): + self.PropName = PropName + self.PropValue = PropValue + + + class Product: + def __init__(self, Id: str = None, Name: str = None, Attributes: List[Attribute] = None): + self.Id = Id + self.Name = Name + + # For each element in this list, + # the VALUE of property 'PropName' will be dynamically indexed. + # e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields. + self.Attributes = Attributes + ``` + + + + ```json + // Sample document content + { + "Name": "SomeName", + "Attributes": [ + { + "PropName": "Color", + "PropValue": "Blue" + }, + { + "PropName": "Width", + "PropValue": "10" + }, + { + "PropName": "Length", + "PropValue": "20" + }, + // ... + ] + } + ``` + + +* **The index**: + + The following index will create a dynamic-index-field per item in the document's `Attributes` list. + New items added to the Attributes list after index creation time will be dynamically indexed as well. + + The actual dynamic-index-field name on which you can query will be the item's PropName **value**. + E.g., 'PropName' value `Width` will be a dynamic-index-field. + + + + ```python + class Attributes_ByName(AbstractIndexCreationTask): + def __init__(self): + super().__init__() + self.map = ( + "from a in docs.Products select new { " + # Define the dynamic-index-fields by calling 'CreateField' + # A dynamic-index-field will be generated for each item in 'Attributes' + + # For each item, the field name will be the value of field 'PropName' + # The field terms will be derived from field 'PropValue' + "_ = a.Attributes.Select(item => CreateField(item.PropName, item.PropValue)), " + + # A regular index field can be defined as well: + "Name = a.Name " + "}" + ) + ``` + + + ```python + class Attributes_ByName_JS(AbstractJavaScriptIndexCreationTask): + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (p) { + return { + // For each item, + // the field name will be the value of field 'PropName' + // The field terms will be derived from field 'PropValue' + _: p.Attributes.map(item => createField(item.PropName, item.PropValue)), + + // A regular index field can be defined as well: + Name: p.Name + }; + }) + """ + ] + ``` + + + +* **The query**: + + To get all documents matching a specific attribute property use: + + + + ```python + matching_documents = list( + session.advanced + .document_query_from_index_type(Attributes_ByName, Product) + # 'Width' is a dynamic-index-field that was indexed from the Attributes list + .where_equals("Width", 10) + ) + ``` + + + ```sql + // 'Width' is a dynamic-index-field that was indexed from the Attributes list + from index 'Attributes/ByName' where Width = 10 + ``` + + + + + + + + + + + +Dynamic fields can be indexed for [Full-text search](../../indexes/querying/searching.mdx) by setting `FieldIndexing.Search` in `CreateFieldOptions`. +* **At indexing time**: + At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms. + The analyzer used at indexing time is resolved from the index definition. + You can configure an analyzer for a specific generated field name, or configure `ALL_FIELDS` as a fallback. + See [Configuring analyzers for dynamic fields](../../indexes/using-dynamic-fields.mdx#configuring-analyzers-for-dynamic-fields) below. + * If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used. + * Otherwise, if `ALL_FIELDS` has an analyzer configuration, RavenDB uses it as a fallback. + * If no analyzer is configured for the field, RavenDB uses the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). -## Indexing documents fields VALUES +* **At querying time**: + For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms. -## Example - basic + When a `search()` query targets a dynamic field that does **not** have an explicit analyzer configuration for its generated field name, + RavenDB uses the [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) to analyze the query term by default. - + Set the [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) + configuration key to `true` to make RavenDB use the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) as the fallback analyzer for the query term instead. -* Only the **basic concept** of creating a dynamic-index-field from the **value** of a document field. -* Documents can then be queried based on those indexed values. + This configuration can be set at different scopes: server-wide, per database, or per index. + The example below sets it at the index scope, in the index definition. + + For a summary of how RavenDB resolves the analyzer used for the query term, + see [Which analyzer is used for the query term](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) below. + +--- + +The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search: + +* **The document**: + + + ```python + class Product: + def __init__(self, Id: str = None, Descriptions: Dict[str, str] = None): + self.Id = Id + self.Descriptions = Descriptions + ``` + + + + ```json + // Sample document content + { + "Descriptions": { + "English": "North wind jacket", + "French": "Veste vent du nord" + } + } + ``` + -* **The document**: - - -{`class Product: - def __init__(self, Id: str = None, product_type: str = None, price_per_unit: float = None): - self.Id = Id - - # The VALUE of ProductType will be dynamically indexed - self.product_type = product_type - self.price_per_unit = price_per_unit -`} - - +* **The index**: - - -{`// Sample document content -\{ - "ProductType": "Electronics", - "PricePerUnit": 23 -\} -`} - - + The following index creates one dynamic field for each entry in the `Descriptions` dictionary. + For example, the key `English` creates the dynamic field `Description_English`. -* **The index**: - The below index will index the **value** of document field 'ProductType'. - - This value will be the dynamic-index-field name on which you can query. - E.g., Field value `Electronics` will be the dynamic-index-field. + Each generated dynamic field is indexed for **full-text search**. + + + + ```python + class Products_ByLocalizedDescription(AbstractIndexCreationTask): + USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = ( + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery" + ) + + def __init__(self): + super().__init__() + self.map = ( + "from product in docs.Products select new { " + " _ = product.Descriptions.Select(x => " + " CreateField(" + " \"Description_\" + x.Key, " + " x.Value, " + # Index each generated dynamic field for FTS + " new CreateFieldOptions { " + " Indexing = FieldIndexing.Search, " + " Storage = FieldStorage.No " + " })) " + "}" + ) + + self._store_all_fields(FieldStorage.NO) + + # Set this configuration key to true so that search() queries on dynamic fields + # will use the 'Default Search Analyzer' when the generated field name + # has no explicit analyzer configuration. + self.configuration = { + Products_ByLocalizedDescription.USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS: "true" + } + ``` + + + ```python + class Products_ByLocalizedDescription_JS(AbstractJavaScriptIndexCreationTask): + USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS = ( + "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery" + ) + + def __init__(self): + super().__init__() + self.maps = [ + """ + map('Products', function (product) { + return { + _: Object.keys(product.Descriptions).map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + // Index each generated dynamic field for FTS + indexing: 'Search', + storage: 'No' + }); + }) + }; + }) + """ + ] + + # Apply a storage default to every generated field + self.fields = { + constants.Documents.Indexing.Fields.ALL_FIELDS: + IndexFieldOptions(storage=FieldStorage.NO) + } + + # Set this configuration key to true so that search() queries on dynamic fields + # will use the 'Default Search Analyzer' when the generated field name + # has no explicit analyzer configuration. + self.configuration = { + Products_ByLocalizedDescription_JS.USE_SEARCH_ANALYZER_FOR_DYNAMIC_FIELDS: "true" + } + ``` + + + +* **Full-text search query**: + + Query the generated dynamic field by its field name. + + In this example, the query targets the generated field `Description_English`. + + Since `Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery` is set to `true`, + the searched term `"north wind"` is analyzed with the [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer). + + + + ```python + results = list( + session.advanced + .document_query_from_index_type(Products_ByLocalizedDescription, Product) + .search("Description_English", "north wind") + ) + ``` + + + + ```sql + from index "Products/ByLocalizedDescription" + where search(Description_English, "north wind") + ``` + + + +--- + + + +### Configuring analyzers for dynamic fields + +`CreateFieldOptions` does **not** let you specify a custom analyzer directly. +However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either: + +* Configuring an analyzer for a specific generated dynamic field name. +* Configuring `ALL_FIELDS` as a fallback for fields that do not have their own explicit analyzer configuration. + +Use `ALL_FIELDS` with care, since it can affect any field that does not have its own explicit analyzer configuration. + +--- + +#### Configure analyzer for a specific dynamic field - -{`class Products_ByProductType(AbstractIndexCreationTask): +```python +class Products_ByLocalizedDescription(AbstractIndexCreationTask): def __init__(self): super().__init__() + self.map = ( + "from product in docs.Products select new { " + " _ = product.Descriptions.Select(x => " + " CreateField(" + " \"Description_\" + x.Key, " + " x.Value, " + " new CreateFieldOptions { " + " Indexing = FieldIndexing.Search, " + " Storage = FieldStorage.No " + " })) " + "}" + ) - # Call 'CreateField' to generate the dynamic-index-fields - # The field name will be the value of document field 'product_type' - # The field terms will be derived from document field 'price_per_unit' - self.map = "from p in docs.Products select new { _ = CreateField(p.product_type, p.price_per_unit)}" -`} - + self._store_all_fields(FieldStorage.NO) + + # Configure an analyzer for a known generated dynamic field name + # ============================================================== + self._analyze("Description_English", "StopAnalyzer") +``` - - -{`class Products_ByProductType_JS(AbstractJavaScriptIndexCreationTask): + +```python +class Products_ByLocalizedDescription_JS(AbstractJavaScriptIndexCreationTask): def __init__(self): super().__init__() - self.maps = { + self.maps = [ """ - map('Products', function (p) { + map('Products', function (product) { return { - _: createField(p.product_type, p.price_per_unit, - { indexing: 'Search', storage: true, termVector: null }) + _: Object.keys(product.Descriptions).map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; }) """ + ] + + # Configure an analyzer for a known generated dynamic field name + # ============================================================== + self.fields = { + constants.Documents.Indexing.Fields.ALL_FIELDS: + IndexFieldOptions(storage=FieldStorage.NO), + "Description_English": + IndexFieldOptions(analyzer="StopAnalyzer") } -`} - - - - -* **The query**: - * To get all documents of some product type having a specific price per unit use: - - - -{`# 'electronics' is the dynamic-index-field that was indexed from the document 'product_type' -matching_documents = list( - session.advanced.document_query_from_index_type(Products_ByProductType, Product).where_equals( - "electronics", 23 - ) -) -`} - - - - -{`// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType' -from index 'Products/ByProductType' where Electronics = 23 -`} - +``` -## Example - list - - - -* Index **values** from items in a list -* After index is deployed, any item added this list in the document will be dynamically indexed as well. - - - -* **The document**: - - -{`class Attribute: - def __init__(self, prop_name: str = None, prop_value: str = None): - self.prop_name = prop_name - self.prop_value = prop_value - - -class Product: - def __init__(self, Id: str = None, name: str = None, attributes: List[Attribute] = None): - self.Id = Id - self.name = name - # For each element in this list, the VALUE of property 'prop_name' will be dynamically indexed - # e.g. color, width, length (in ex. below) will become dynamic-index-field - self.attributes = attributes -`} - - - - - -{`// Sample document content -\{ -Name": "SomeName", -Attributes": [ - \{ - "PropName": "Color", - "PropValue": "Blue" - \}, - \{ - "PropName": "Width", - "PropValue": "10" - \}, - \{ - "PropName": "Length", - "PropValue": "20" - \}, - ... - -\} -`} - - - -* **The index**: - The below index will create a dynamic-index-field per item in the document's `Attributes` list. - New items added to the Attributes list after index creation time will be dynamically indexed as well. - - The actual dynamic-index-field name on which you can query will be the item's PropName **value**. - E.g., 'PropName' value `width` will be a dynamic-index-field. +#### Configure a fallback analyzer for all fields - -{`class Attributes_ByName(AbstractIndexCreationTask): +```python +class Products_ByLocalizedDescription(AbstractIndexCreationTask): def __init__(self): super().__init__() self.map = ( - "from a in docs.Products select new " - "{ _ = a.attributes.Select( item => CreateField(item.prop_name, item.prop_value)), name = a.name " + "from product in docs.Products select new { " + " _ = product.Descriptions.Select(x => " + " CreateField(" + " \"Description_\" + x.Key, " + " x.Value, " + " new CreateFieldOptions { " + " Indexing = FieldIndexing.Search, " + " Storage = FieldStorage.No " + " })) " "}" ) -`} - + + self._store_all_fields(FieldStorage.NO) + + # Use this as a fallback for fields that do not have + # their own explicit analyzer configuration + # ================================================== + self._analyze(constants.Documents.Indexing.Fields.ALL_FIELDS, "StopAnalyzer") +``` - - -{`class Attributes_ByName_JS(AbstractJavaScriptIndexCreationTask): + +```python +class Products_ByLocalizedDescription_JS(AbstractJavaScriptIndexCreationTask): def __init__(self): super().__init__() - self.maps = { + self.maps = [ """ - map('Products', function (p) { + map('Products', function (product) { return { - _: p.Attributes.map(item => createField(item.PropName, item.PropValue, - { indexing: 'Search', storage: true, termVector: null })), - Name: p.Name + _: Object.keys(product.Descriptions).map(function (key) { + return createField( + 'Description_' + key, + product.Descriptions[key], + { + indexing: 'Search', + storage: 'No' + }); + }) }; }) """ + ] + + # Use this as a fallback for fields that do not have + # their own explicit analyzer configuration + # ================================================== + self.fields = { + constants.Documents.Indexing.Fields.ALL_FIELDS: + IndexFieldOptions(storage=FieldStorage.NO, analyzer="StopAnalyzer") } -`} - - - - -* **The query**: - To get all documents matching a specific attribute property use: - - - -{`matching_documents = list( - session.advanced.document_query_from_index_type(Attributes_ByName, Product).where_equals( - "width", 10 - ) -) -`} - - - - -{`// 'Width' is a dynamic-index-field that was indexed from the Attributes list -from index 'Attributes/ByName' where Width = 10 -`} - +``` + + + + +### Which analyzer is used for the query term? + +The following applies when a `search()` query targets a **dynamic field**: + +| Index configuration | [Configuration key](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) | Analyzer used for the query term | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| No analyzer is explicitly configured for the generated field name | `false` | The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) | +| No analyzer is explicitly configured for the generated field name | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `false` | The analyzer resolved by the normal field fallback path,
e.g. `WhitespaceAnalyzer` | +| `ALL_FIELDS` is configured with a specific analyzer,
e.g. `WhitespaceAnalyzer` | `true` | The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) | +| The generated field name is explicitly configured,
e.g. `self._analyze("Description_English", "StopAnalyzer")` | `false` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | +| The generated field name is explicitly configured,
e.g. `self._analyze("Description_English", "StopAnalyzer")` | `true` | The explicitly configured analyzer,
e.g. `StopAnalyzer` | + + + -## CreateField syntax + #### Syntax for Index: - - -{`object CreateField(string name, object value); + +```python +CreateField(name, value) -object CreateField(string name, object value, bool stored, bool analyzed); +CreateField(name, value, stored, analyzed) -object CreateField(string name, object value, CreateFieldOptions options); -`} - +CreateField(name, value, CreateFieldOptions) +``` #### Syntax for JavaScript-index: - - -{`createField(fieldName, fieldValue, options); // returns object -`} - + +```js +createField(fieldName, fieldValue, options); // returns object +``` -| Parameters | Type | Description | -|------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **name** | `string` | Name of the dynamic-index-field | -| **value** | `object` | Value of the dynamic-index-field
The field Terms are derived from this value. | -| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`False` - will set `FieldStorage.NO` (default value)
`True` - will set `FieldStorate.YES` | -| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`None` - `FieldIndexing.Default` (default value)
`False` - `FieldIndexing.Exact`
`True` - `FieldIndexing.Search` | -| **options** | `CreateFieldOptions` | Dynamic-index-field options | +| Parameters | Type | Description | +|----------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **fieldName** | `string` | Name of the dynamic-index-field | +| **fieldValue** | `object` | Value of the dynamic-index-field
The field terms are derived from this value. | +| **stored** | `bool` | Sets [FieldStorage](../../indexes/storing-data-in-index.mdx)

`false` - will set `FieldStorage.No` (default value)
`true` - will set `FieldStorage.Yes` | +| **analyzed** | `bool` | Sets [FieldIndexing](../../indexes/using-analyzers.mdx)

`null` - `FieldIndexing.Default` (default value)
`false` - `FieldIndexing.Exact`
`true` - `FieldIndexing.Search` | +| **options** | `CreateFieldOptions` | Dynamic-index-field options | -| CreateFieldOptions | | | -|--------------------|--------------------|----------------------------------------------------------------------------| +| CreateFieldOptions | | | +|--------------------|--------------------|-----------------------------------------------------------------------------------| | **Storage** | `FieldStorage?` | Learn about [storing data](../../indexes/storing-data-in-index.mdx) in the index. | -| **Indexing** | `FieldIndexing?` | Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | -| **TermVector** | `FieldTermVector?` | Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | +| **Indexing** | `FieldIndexing?` | Sets the field indexing behavior.
Learn about [using analyzers](../../indexes/using-analyzers.mdx) in the index. | +| **TermVector** | `FieldTermVector?` | Sets the field term-vector behavior.
Learn about [term vectors](../../indexes/using-term-vectors.mdx) in the index. | @@ -496,17 +892,15 @@ object CreateField(string name, object value, CreateFieldOptions options); + + + - -## Indexed fields & terms view - -The generated dynamic-index-fields and their indexed terms can be viewed in the **Terms View**. -Below are sample index fields & their terms generated from the last example. +The generated dynamic index fields and their indexed terms can be viewed in the **Terms View** in Studio. +Below are sample index fields & their terms generated from [this example](../../indexes/using-dynamic-fields.mdx#example---list). ![Figure 1. Go to terms view](../assets/dynamic-index-fields-1.png) ![Figure 2. Indexed fields & terms](../assets/dynamic-index-fields-2.png) - - - - + + diff --git a/versioned_docs/version-7.1/indexes/querying/content/_searching-csharp.mdx b/versioned_docs/version-7.1/indexes/querying/content/_searching-csharp.mdx index d20ef5a65e..f0be0e6ffd 100644 --- a/versioned_docs/version-7.1/indexes/querying/content/_searching-csharp.mdx +++ b/versioned_docs/version-7.1/indexes/querying/content/_searching-csharp.mdx @@ -16,11 +16,13 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) * [Searching with wildcards](../../../indexes/querying/searching.mdx#searching-with-wildcards) * [When using RavenStandardAnalyzer or StandardAnalyzer or NGramAnalyzer](../../../indexes/querying/searching.mdx#when-usingoror) @@ -28,6 +30,7 @@ import CodeBlock from '@theme/CodeBlock'; * [When using the Exact analyzer](../../../indexes/querying/searching.mdx#when-using-the-exact-analyzer) + ## Indexing single field for FTS #### The index: @@ -262,7 +265,7 @@ where (search(EmployeeData, "Manager") or search(EmployeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method in the _Map_ function to extract all property values and index them in a single searchable field. * This approach makes the index robust to changes in the document schema. @@ -377,7 +380,14 @@ where search(AllValues, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). ## Boosting search results diff --git a/versioned_docs/version-7.1/indexes/querying/content/_searching-java.mdx b/versioned_docs/version-7.1/indexes/querying/content/_searching-java.mdx index 7d85246ced..4accf3f512 100644 --- a/versioned_docs/version-7.1/indexes/querying/content/_searching-java.mdx +++ b/versioned_docs/version-7.1/indexes/querying/content/_searching-java.mdx @@ -388,4 +388,11 @@ where search(allValues, "tofu") +## Indexing dynamic fields for FTS +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). \ No newline at end of file diff --git a/versioned_docs/version-7.1/indexes/querying/content/_searching-nodejs.mdx b/versioned_docs/version-7.1/indexes/querying/content/_searching-nodejs.mdx index 4ca66b5877..cb73bfd7c0 100644 --- a/versioned_docs/version-7.1/indexes/querying/content/_searching-nodejs.mdx +++ b/versioned_docs/version-7.1/indexes/querying/content/_searching-nodejs.mdx @@ -5,7 +5,7 @@ import CodeBlock from '@theme/CodeBlock'; -* Prior to reading this article, please refer to [full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) +* Prior to reading this article, please refer to [Full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) to learn about the `search` method. * **All capabilities** provided by `search` with a dynamic query can also be used when querying a static-index. @@ -17,11 +17,13 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) * [Searching with wildcards](../../../indexes/querying/searching.mdx#searching-with-wildcards) * [When using RavenStandardAnalyzer or StandardAnalyzer or NGramAnalyzer](../../../indexes/querying/searching.mdx#when-usingoror) @@ -29,6 +31,7 @@ import CodeBlock from '@theme/CodeBlock'; * [When using the Exact analyzer](../../../indexes/querying/searching.mdx#when-using-the-exact-analyzer) + ## Indexing single field for FTS #### The index: @@ -163,7 +166,7 @@ where (search(employeeData, "Manager") or search(employeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, which is available when using **a C# LINQ string** that is assigned to the `map` property in the Node.js index class, as shown in the example below. @@ -229,6 +232,14 @@ where search(AllValues, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). ## Boosting search results diff --git a/versioned_docs/version-7.1/indexes/querying/content/_searching-php.mdx b/versioned_docs/version-7.1/indexes/querying/content/_searching-php.mdx index 55261ce19f..510a6f8752 100644 --- a/versioned_docs/version-7.1/indexes/querying/content/_searching-php.mdx +++ b/versioned_docs/version-7.1/indexes/querying/content/_searching-php.mdx @@ -5,7 +5,7 @@ import CodeBlock from '@theme/CodeBlock'; -* Prior to reading this article, please refer to [full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) +* Prior to reading this article, please refer to [Full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) to learn about the `search` method. * **All capabilities** provided by `search` with a dynamic query can also be used when querying a static-index. @@ -17,14 +17,17 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) + ## Indexing single field for FTS #### The index: @@ -253,7 +256,7 @@ where (search(EmployeeData, "Manager") or search(EmployeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, which is available in the **C# LINQ string** that is assigned to the `map` property in the PHP index class, as shown in the example below. @@ -332,6 +335,15 @@ where search(allValues, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). + ## Boosting search results diff --git a/versioned_docs/version-7.1/indexes/querying/content/_searching-python.mdx b/versioned_docs/version-7.1/indexes/querying/content/_searching-python.mdx index f06c29d94f..88c6aa452f 100644 --- a/versioned_docs/version-7.1/indexes/querying/content/_searching-python.mdx +++ b/versioned_docs/version-7.1/indexes/querying/content/_searching-python.mdx @@ -5,7 +5,7 @@ import CodeBlock from '@theme/CodeBlock'; -* Prior to reading this article, please refer to [full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) +* Prior to reading this article, please refer to [Full-Text search with dynamic queries](../../../client-api/session/querying/text-search/full-text-search.mdx) to learn about the `search` method. * **All capabilities** provided by `search` with a dynamic query can also be used when querying a static-index. @@ -17,14 +17,17 @@ import CodeBlock from '@theme/CodeBlock'; See examples below. * You can configure which analyzer will be used to tokenize this field. - See [selecting an analyzer](../../../indexes/using-analyzers.mdx#selecting-an-analyzer-for-a-field). + Learn more in [Setting analyzer for index-field](../../../indexes/using-analyzers.mdx#setting-analyzer-for-index-field). + * In this article: * [Indexing single field for FTS](../../../indexes/querying/searching.mdx#indexing-single-field-for-fts) * [Indexing multiple fields for FTS](../../../indexes/querying/searching.mdx#indexing-multiple-fields-for-fts) - * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-(using-asjson)) + * [Indexing all fields for FTS (using AsJson)](../../../indexes/querying/searching.mdx#indexing-all-fields-for-fts-using-asjson) + * [Indexing dynamic fields for FTS](../../../indexes/querying/searching.mdx#indexing-dynamic-fields-for-fts) * [Boosting search results](../../../indexes/querying/searching.mdx#boosting-search-results) + ## Indexing single field for FTS #### The index: @@ -167,7 +170,7 @@ where (search(EmployeeData, "Manager") or search(EmployeeData, "French Spanish", ## Indexing all fields for FTS (using AsJson) -* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, +* To search across ALL fields in a document without defining each one explicitly, use the `AsJson` method, which is available in the **C# LINQ string** that is assigned to the `map` property in the Python index class, as shown in the example below. @@ -238,6 +241,14 @@ where search(all_values, "tofu") +## Indexing dynamic fields for FTS + +* [Dynamic index fields](../../../indexes/using-dynamic-fields.mdx) are created at indexing time, with names and values derived from document content, + rather than being predefined as fixed index fields in the index definition. + This allows you to query the index by fields that are not known at index definition or deployment time. + +* Dynamic index fields can be configured for full-text search just like regular index fields. + Learn more in [Indexing dynamic fields for full-text search](../../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search). ## Boosting search results diff --git a/versioned_docs/version-7.1/indexes/using-dynamic-fields.mdx b/versioned_docs/version-7.1/indexes/using-dynamic-fields.mdx index 7e6fe3a7ca..e465ba5431 100644 --- a/versioned_docs/version-7.1/indexes/using-dynamic-fields.mdx +++ b/versioned_docs/version-7.1/indexes/using-dynamic-fields.mdx @@ -1,6 +1,7 @@ --- -title: "Indexes: Dynamic Index Fields" -sidebar_label: Dynamic Fields +title: "Dynamic Index Fields" +sidebar_label: "Dynamic Index Fields" +description: "Index document fields dynamically in RavenDB when field names are unknown at index definition time, using the CreateField method." sidebar_position: 27 supported_languages: ["csharp", "java", "python", "php", "nodejs"] see_also: @@ -31,7 +32,6 @@ import UsingDynamicFieldsPython from './content/_using-dynamic-fields-python.mdx import UsingDynamicFieldsPhp from './content/_using-dynamic-fields-php.mdx'; import UsingDynamicFieldsNodejs from './content/_using-dynamic-fields-nodejs.mdx'; - diff --git a/versioned_docs/version-7.1/server/configuration/indexing-configuration.mdx b/versioned_docs/version-7.1/server/configuration/indexing-configuration.mdx index 2e1f67f6a6..3ff0ff574f 100644 --- a/versioned_docs/version-7.1/server/configuration/indexing-configuration.mdx +++ b/versioned_docs/version-7.1/server/configuration/indexing-configuration.mdx @@ -103,6 +103,7 @@ import LanguageContent from "@site/src/components/LanguageContent"; [Indexing.OrderByTicksAutomaticallyWhenDatesAreInvolved](../../server/configuration/indexing-configuration.mdx#indexingorderbyticksautomaticallywhendatesareinvolved) [Indexing.QueryClauseCache.Disabled](../../server/configuration/indexing-configuration.mdx#indexingqueryclausecachedisabled) [Indexing.QueryClauseCache.RepeatedQueriesTimeFrameInSec](../../server/configuration/indexing-configuration.mdx#indexingqueryclausecacherepeatedqueriestimeframeinsec) + [Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery](../../server/configuration/indexing-configuration.mdx#indexingqueryingusesearchanalyzerfordynamicfieldsifnotsetexplicitlyinsearchquery) [Indexing.ScratchSpaceLimitInMb](../../server/configuration/indexing-configuration.mdx#indexingscratchspacelimitinmb) [Indexing.Static.SearchEngineType](../../server/configuration/indexing-configuration.mdx#indexingstaticsearchenginetype) [Indexing.Throttling.TimeIntervalInMs](../../server/configuration/indexing-configuration.mdx#indexingthrottlingtimeintervalinms) @@ -1008,6 +1009,33 @@ Queries that repeat within this time frame will be considered worth caching. +## Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery + +* This setting controls which analyzer is used for the **query term** when a **`search()` query** targets a [Dynamic index field](../../indexes/using-dynamic-fields.mdx#indexing-dynamic-fields-for-full-text-search) + that was indexed for full-text search, and no analyzer is explicitly configured for that field in the index definition. + + * `false` (default): + The [Default analyzer](../../indexes/using-analyzers.mdx#using-the-default-analyzer) is used to analyze the **query term**. + This analyzer can be configured via the [Indexing.Analyzers.Default](../../server/configuration/indexing-configuration.mdx#indexinganalyzersdefault) configuration key. + + * `true`: + The [Default search analyzer](../../indexes/using-analyzers.mdx#using-the-default-search-analyzer) is used to analyze the **query term**. + This analyzer can be configured via the [Indexing.Analyzers.Search.Default](../../server/configuration/indexing-configuration.mdx#indexinganalyzerssearchdefault) configuration key. + +* See the table in [Which analyzer is used for the query term?](../../indexes/using-dynamic-fields.mdx#which-analyzer-is-used-for-the-query-term) + for the complete analyzer-selection rules based on this configuration key and the analyzer configured for the dynamic index field. + +* This configuration applies to both the Lucene and Corax search engines. + The default is `false` to preserve backward compatibility. + +--- + +- **Type**: `bool` +- **Default**: `false` +- **Scope**: Server-wide, or per database, or per index + + + ## Indexing.ScratchSpaceLimitInMb * Amount of scratch space in megabytes that we allow to use for the index storage.