Many resources are needed to download a project. Please understand that we have to compensate our server costs. Thank you in advance. Project price only 1 $
You can buy this project and download/modify it how often you want.
/*
* Copyright (c) 2022 nosqlbench
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.nosqlbench.engine.api.templating;
import io.nosqlbench.api.config.standard.NBConfigError;
import io.nosqlbench.engine.api.activityconfig.yaml.OpTemplate;
import io.nosqlbench.engine.api.templating.binders.ArrayBinder;
import io.nosqlbench.engine.api.templating.binders.ListBinder;
import io.nosqlbench.engine.api.templating.binders.OrderedMapBinder;
import io.nosqlbench.api.config.fieldreaders.DynamicFieldReader;
import io.nosqlbench.api.config.fieldreaders.StaticFieldReader;
import io.nosqlbench.api.config.standard.NBConfiguration;
import io.nosqlbench.api.errors.OpConfigError;
import io.nosqlbench.virtdata.core.templates.BindPoint;
import io.nosqlbench.virtdata.core.templates.CapturePoint;
import io.nosqlbench.virtdata.core.templates.ParsedStringTemplate;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.util.*;
import java.util.function.BiFunction;
import java.util.function.Function;
import java.util.function.LongFunction;
/**
*
ParsedOp API
*
*
This is the primary developer-focused API for driver developers to use when up-converting op templates
* to operations. This {@link ParsedOp} is a wrapper around the op template structure. It provides many
* ways of constructing higher-order objects from a variety of sources.
*
*
Supporting Variety
*
* For some drivers or protocols, the primary user interface is a statement format or grammar. For CQL or SQL,
* the most natural way of describing templates for operations is in that native format. For others, an operation
* template may look like a block of JSON or a HTTP request. All these forms are supported. In order to deal with
* the variety, there is a set of detailed rules for how the workload definitions are transformed into driver
* operations for a native driver. The high-level flow is:
*
*
Op Template Form
*
Normalized Data Structure
*
ParsedOp Form
*
*
* The specifications govern how the raw op template form is transformed into the normalized data structure.
* This documentation focuses on the API provided by the last form, this class, although peripheral awareness
* of the first two forms will certainly help for any advanced scenarios. You can find detailed examples and
* specification tests under the workload_definition folder of the adapters-api module.
*
*
*
*
Op Template Parsing
*
*
Rule #1: All op templates are parsed into an internal normalized structure which contains:
*
*
A map of op fields, which can consist of:
*
*
static field values
*
dynamic field values, the actual value of which can only be known for a given cycle
*
*
*
Access to auxiliary configuration values like activity parameters. These can back-fill
* when values aren't present in the direct static or dynamic op fields.
*
*
*
Rule #2: When asking for a dynamic parameter, static parameters may be automatically promoted to functional form as a back-fill.
*
Rule #3: When asking for static parameters, config parameters may automatically be promoted as a back-fill.
*
* The net effect of these rules is that the NoSQLBench driver developer may safely use functional forms to access data
* in the op template, or may decide that certain op fields must only be provided in a static way per operation.
*
*
*
Distinguishing Op Payload from Op Config
*
When a user specifies an op template, they may choose to provide only a single set of op fields without
* distinguishing between config or payload, or they may choose to directly configure each.
*
Example:
*
{@code
* ops:
* # both op and params explicitly named
* op1:
* op:
* opfield1: value1
* params:
* param2: value2
* # neither op field nor params named, so all assumed to be op fields
* op2:
* opfield1: value1
* param2: value2
* # in this case, if param2 is meant to be config level,
* # it is a likely config error that should be thrown to the user
* # only op field explicitly named, so remainder automatically pushed into params
* op3:
* op:
* opfield1: value1
* param2: value2
* # only params explicitly named, so remainder pushed into op payload
* op4:
* params:
* param2: value2
* opfield1: value1
* }
*
* All of these are considered valid constructions, and all of them may actually achieve the same result.
* This looks like an undesirable problem, but it serves to simplify things for users in one specific way: It allows
* them to be a vague, within guidelines, and still have a valid workload definition.
* The NoSQLBench runtime does a non-trivial amount of processing on the op template to
* ensure that it can conform to an unambiguous normalized internal structure on your behalf.
* This is needed because of how awful YAML is as a user configuration language in spite of its
* ubiquity in practice. The basic design guideline for these forms is that the op template must
* mean what a reasonable user would assume without looking at any documentation.
*
*
*
Design Invariants
*
*
The above rules imply invariants, which are made explicit here. {@link ParsedOp}.
*
*
*
*
You may not use an op field name or parameter name for more than one purpose.
*
*
Treat all parameters supported by a driver adapter and it's op fields as a globally shared namespace, even if it is not.
* This avoids creating any confusion about what a parameter can be used for and how to use it for the right thing in the right place.
* For example, you may not use the parameter name `socket` in an op template to mean one thing and then use it
* at the driver adapter level to mean something different. However, if the meaning is congruent, a driver developer
* may choose to support some cross-cutting parameters at the activity level. These allowances are explicit,
* however, as each driver dictates what it will allow as activity parameters.
*
*
*
*
*
Users may specify op payload fields within op params or activity params as fallback config sources in that order.
*
*
IF a name is valid as an op field, it must also be valid as such when specified in op params.
*
If a name is valid as an op field, it must also be valid as such when specified in activity params, within the scope of {@link ParsedOp}
*
When an op field is found via op params or activity params, it may NOT be dynamic. If dynamic values are intended to be provided
* at a common layer in the workload, then bindings support this already.
*
*
*
*
You must access non-payload params via Config-oriented methods.
*
*
Op Templates contain op payload data and op configs (params, activity params).
*
You must use only {@link ParsedOp} getters with "...Config..." names, such as {@link #getConfigOr(String, Object, long)}
* when accessing non-payload fields.
*
When a dynamic value is found via one of these calls, an error should be thrown,
* as configuration level data is not expected to be variant per-operation or cycle.
*
*
*
*
The user must be warned when a required or optional config value is missing from op params (or activity params), but a value
* of the same name is found in op payload fields.
*
*
If rule #1 is followed, and names are unambiguous across the driver, then it is almost certainly a configuration error.
*
*
*
*
When both an op payload field and a param field of the same name are defined through cascading configuration of param fields,
* the local op payload field takes precedence.
*
*
This is an extension of the param override rules which say that the closest (most local) value to an operation is the one that takes precedence.
*
In practice, there will be no conflicts between direct static and dynamic fields, but there will be possibly between
* static or dynamic fields and parameters and activity params. If a user wants to promote an activity param as an override to existing op fields,
* template variables allow for this to happen gracefully. Otherwise, the order of precedence is 1) op fields 2) op params 3) activity params.
*
*
*
*
*
*
*
*
Op Payload Forms
* Field values can come from multiple sources. These forms and any of their combinations are supported.
*
*
*
* As shown, any literal value of any valid YAML type, including structured values like lists or maps are accepted as
* static op template values. A static value is any value which contains zero bind points at any level.
*
*
In this form, {@code {binding1}} is known as a binding reference, since the binding function is defined
* elsewhere under the given name. The first field "field1" is specified with no leading nor trailing literals, and
* is thus taken as a raw binding reference, meaning that it will not be converted to a String. The second,
* named "field2", is what is known as a string template, and is syntactical sugar for a more complex binding
* which concatenates static and dynamic parts together. In this form, object types produced by binding functions are
* converted to string form before concatenation.
*
*
*
Note that a raw {@code {binding1}} value (without quotes) would be NOT be a binding reference, since YAML
* is a superset of JSON. this means that {@code {binding1}} would be converted to a map or JSON object type
* with invalid contents. This is warned about when detected as a null valued map key, although it also makes
* null values invalid for ANY op template value.
This example combines the previous ones with structure and dynamic values. Both field1 and field2 are dynamic,
* since each contains some dynamic value or template within. When field1 is accessed within a cycle, that cycle's value
* will be used as the seed to generate equivalent structures with all the literal and dynamic elements inserted as
* the template implies. As before, direct binding references like {@code {binding4}} will be inserted into the
* structure with whatever type the binding definition produces, so if you want strings in them, ensure that you
* configure your binding definitions thusly.
*
* The params section are the first layer of external configuration values that an op template can use to distinguish
* op configuration parameters from op payload content.
* Op Template Params are referenced when any of the {@link #getConfigOr(String, Object, long)} or other ...Config...
* getters are used (bypassing non-param fields). They are also accessed as a fallback when no static nor dynamic value is found
* for a reference op template field. Unlike op fields, op params cascade from within the workload YAML from the document level,
* down to each block and then down to each statement.
*
*
Activity Params
*
Example:
*
{@code
* ./nb run driver=... workload=... cl=LOCAL_QUORUM
* }
*
*
When explicitly allowed by a driver adapter's configuration model, values like {@code cl} above can be seen as
* another fallback source for configuration parameters. The {@link ParsedOp} implementation will automatically look
* in the activity parameters if needed to find a missing configuration parameter, but this will only work if
* the specific named parameter is allowed at the activity level.