se.config.advanced-configuration.adoc Maven / Gradle / Ivy
The newest version!
///////////////////////////////////////////////////////////////////////////////
Copyright (c) 2018, 2024 Oracle and/or its affiliates.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
///////////////////////////////////////////////////////////////////////////////
= Advanced Configuration Topics
:description: Helidon advanced configuration
:keywords: helidon, config, meta
:feature-name: Config
:rootdir: {docdir}/../..
include::{rootdir}/includes/se.adoc[]
== Contents
- <>
- <>
- <>
- <>
- <>
- <>
== Overview
This section discusses several advanced topics related to Helidon configuration.
[[_advanced_config_sources]]
== Advanced Config Sources
=== Environment Variables Config Source
The config system supports using environment variables as a config source, and is
enabled by default. Since environment variable names are normally restricted to
alphanumeric characters and underscore, this config source _adds_ aliases that
enable setting or overriding config entries with dotted and/or hyphenated keys.
The mapping makes it possible to set or override a config entry with a key of
`"foo.bar"` using an environment variable named `"FOO_BAR"` and `"foo.bar-baz"`
using `"FOO_BAR_dash_BAZ"`.
One use case for this mapping is config overrides in containers, where passing
environment variables directly or via Kubernetes Secrets/ConfigMaps is common.
Scripts that solve the mapping problem by explicitly converting variables to
system properties can also be simplified.
Aliases are produced for any environment variable name that matches _all_ the following:
. does not begin or end with a `'_'` character
. does not contain `"__"`
. contains one or more `'_'` characters
For each such name, two aliases are added with the names mapped as follows:
. Replace any `"\_dash_"` or `"\_DASH_"` substrings with `"-"`, e.g. `"APP_PAGE_dash_SIZE"`
becomes `"APP_PAGE-SIZE"`.
. Replace `'_'` with `'.'` and add as an alias, e.g. `"APP_GREETING"` is
added as `"APP.GREETING"` and `"APP_PAGE-SIZE"` is added as `"APP.PAGE-SIZE"`.
This mapping is added primarily to support mixed case config keys such as
`"app.someCamelCaseKey"`.
. Convert the result of step 2 to lowercase and add as an alias, e.g.
`"APP.GREETING"` is added as `"app.greeting"` and `"APP.PAGE-SIZE"` is added
as `"app.page-size"`.
=== Directory Config Source
The config system supports using a file system directory as a config source.
Each _non-directory_ file in the directory becomes a config entry: the file name
is the key and the contents of that file
are used as the corresponding config `String` value.
The following example shows, for example, one way to load Kubernetes secrets
mounted on the pod's filesystem.
If the directory `conf/secrets` contains these two files
[source]
.File `secrets/username`
----
jose
----
[source]
.File `secrets/password`
----
^ery$ecretP&ssword
----
your application can load this as configuration as follows:
[source,java]
.Using `directory` config source
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_1, indent=0]
----
<1> Loads all files from the `conf/secrets` directory.
<2> No need to use environment variables or system properties as sources in building
the `Config`.
<3> The loaded config maps the key `username` to the value `jose`...
<4> ...and the key `password` to `^ery$ecretP&ssword`.
Remember that your application can process the contents of a given file
as configuration. See the xref:introduction.adoc#config_sources[config sources] section
and the link:{config-javadoc-base-url}/io/helidon/config/ConfigSources.html#file-java.lang.String-[`ConfigSources.file`]
JavaDoc.
=== In-memory Config Sources
The config system provides several ways to create a `Config` tree from data
already in memory. See the link:{config-javadoc-base-url}/io/helidon/config/ConfigSources.html[`ConfigSources` javadoc]
for further details. The numerous variants of the `from` method construct
`ConfigSource` or `Builder` instances.
==== Subtree of Another `Config`
[source,java]
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_2, indent=0]
----
==== `Properties` Object
[source,java]
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_3, indent=0]
----
==== `String` of a Given Media Type
[source,java]
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_4, indent=0]
----
==== `Map`
[source,java]
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_5, indent=0]
----
==== _ad hoc_ Config Nodes
[source,java]
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_6, indent=0]
----
<1> `ConfigSources.create` variants for `Properties` or `Map` arguments return a
`MapConfigSource.Builder` instance.
=== Multi-Source Configs and Composite Config Sources
Although the examples above use a single source, you can build a single `Config`
from multiple sources.
==== Handling Key Collisions
===== Prefixed Config Sources
Sometimes you might want to create a single config tree from
multiple sources but in a way that keeps the config from different sources
in different subtrees.
The config system lets you assign a prefix to all keys
from a given source using the
link:{config-javadoc-base-url}/io/helidon/config/ConfigSources.html#prefixed-java.lang.String-java.util.function.Supplier-[`ConfigSources.prefixed`] method.
The following example shows two YAML files as config sources
and the code to load each with a different prefix into a single `Config` tree:
[source,hocon]
.File `app.conf`
----
greeting = "Hello"
page-size = 20
basic-range = [ -20, 20 ]
----
[source,hocon]
.File `data.conf`
----
providers: [
{
name = "Provider1"
class = "this.is.my.Provider1"
},
{
name = "Provider2"
class = "this.is.my.Provider2"
}
]
----
[source,java]
.Using `prefixed` config source
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_7, indent=0]
----
<1> Specifies the prefix `app` for the associated source.
<2> `Supplier` for the file
`app.conf` loaded from the current `classpath`.
<3> Specifies the prefix `data` for the associated source.
<4> Supplier for the file `app.conf` loaded from the current `classpath`.
<5> Key `app.greeting` combines the `app` prefix and the original key `greeting`
from the `app.conf` source.
<6> Key `data.providers.0.name` combines the `data` prefix and
the original key `providers.0.name` property from `data.conf` source.
This technique can be useful, for example, if multiple sources contain
keys that might overlap; assigning different prefixes to the keys from different
sources gives your application a way to access all config elements distinctly even
if their keys would otherwise conflict.
===== Merging Strategies
When creating config from multiple sources, it is possible that the same key comes from multiple
sources. By default, earlier sources in the list have higher priority than later ones. This means
that if the same key appears in two or more sources, then the source earlier in the list prevails.
The config system provides the
`FallbackMergingStrategy`
which implements the default, "first wins" algorithm. You can write your own
implementation of MergingStrategy interface
and use it instead to provide a different algorithm.
[source,java]
.Composite config source example
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_8, indent=0]
----
<1> Specifies the merging strategy. This example uses the default fallback
merging strategy.
[[_advanced_config_parsers]]
== Advanced Config Parsers
Config sources and parsers work together to read and translate configuration data from some
external form into the corresponding in-memory config tree.
=== How Config Chooses Parsers [[Config-Advanced-Sources-SuitableParser]]
Although most applications are explicit about the config sources they use in building a `Config`, the config system often has to figure out what parser to use. It does so by:
1. determining, the best that it can, the media type of the source, and
2. locating a parser that can translate that media type.
==== Identifying the Media Type
===== By Inference
Most applications let the config system try to infer the media type of the
config source.
By default, config source implementations use the
`io.helidon.common.media.type.MediaTypes` API to infer the source media type from
the source, typically (but not always) based on the file type portion of the file path.
Helidon media type module has a predefined set of mappings as configured in
`common/media-type/src/main/resources/io/helidon/common/media/type/default-media-types.properties`, including
the Config supported formats: `.properties`, `.yaml`, `.json` and `.conf`. To handle
other formats you can implement and register your own `io.helidon.common.media.type.spi.MediaTypeDetector` Java Service
implementations. (Typically, you would also write and register a config parser
to translate that format; see <> below.)
===== By Application Directive
Your application can specify what media type to use in interpreting a config
source. Use this if your application knows the media type but the system might
not be able to infer it correctly, either because no type detector would recognize it or
because there might be more than one inferred media type.
[source,java]
.Specify `mediaType` for config source
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_9, indent=0]
----
<1> The config system cannot infer the media type because there is no file
type in the path `props`.
<2> The developer knows the file is in Java Properties format so specifies the
media type explicitly.
Note that a file type detector _could_ be written to
also inspect the contents of the file to infer the media type. The detectors
provided by Helidon only inspect the suffix in the name of the file.
==== Locating a Parser [[locating-parser]]
===== By Inference from `media-type`
Each config parser reports which media types it handles. Once the config system
has determined a source's media type, it searches the config parsers associated
with the config builder for one that recognizes that media type. It then uses
that parser to translate the config in the source into the in-memory config tree.
The application can add one or more parsers to a `Config.Builder`
using the `addParser` method. This makes the parser available for use by the
config sources associated with that builder, but does not directly tie a given
parser to a given source. The builder uses media-type matching to select one of
the parsers registered with the builder for each source.
If the config system cannot locate a parser that matches the media-type of a source, it throws
a `ConfigException` when trying to prepare the configuration.
===== By Application Directive
Your application can specify which parser to use for a config source. The
`AbstractConfigSourceBuilder` class exposes the `parser` method, which
accepts the `ConfigParser` to be used for that source. Several methods
on `ConfigSources` such as `classpath`, `directory`, and `file` return this
builder class.
Generally try to rely on media-type matching rather than specifying a given parser
for a given source in the application. This keeps your application more flexible,
both by insulating it from implementation classes and by letting it easily take
advantage of improvements in or alternatives to the parsers available for a given
media type.
[source,java]
.Specify `parser` for config source
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_10, indent=0]
----
<1> The config system cannot infer the media type because there is no file
type in the path `props`.
<2> The developer knows the file is in Java Properties format so specifies the
properties parser explicitly.
=== Parsing a Config Value as Config
A config value node might contain an entire config document in `String` form, but in
a format different from the containing document. Your application can tell the
config system to parse such a node as config in a different format and replace
the `String` value node in the original tree with the config tree that results
from parsing that `String`.
In this example, a YAML document contains a JSON document as a leaf.
[source,yaml]
.YAML file with included JSON formatted property
----
secrets:
username: "jose"
password: "^ery$ecretP&ssword"
app: > # <1>
{
"greeting": "Hello",
"page-size": 20,
"basic-range": [ -20, 20 ]
}
----
<1> The property `app` is itself formatted as a JSON document.
==== Specify Key-to-media-type Mapping
[source,java]
.Specify JSON as media type for node
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_11, indent=0]
----
<1> The source builder's `mediaTypeMapping` method accepts a function
which returns the appropriate media types (if any) for config keys.
<2> The function says to treat the `app` property value as a JSON document and
leave other nodes unchanged.
<3> Other properties are loaded as expected.
<4> Property `app` is now a structured object node.
Because the function passed to `mediaTypeMapping` identifies the `app` node as a JSON
document, the config system selects the config parser that is registered with the builder
which also handles the JSON media type.
Also, note that the config system replaces the original `String` value node with
an object node resulting from parsing that `String` value as JSON.
==== Specify Key-to-parser Mapping
Alternatively, your application could map config keys to the specific parsers
you want to use for parsing those keys' values.
[source,java]
.Specify JSON formatted property' parser instance
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_12, indent=0]
----
<1> Uses the `parserMapping` method to map keys to parser instances.
<2> Tells the config system to use the HOCON parser for translating the `String`
value of the `app` key. (HOCON is a superset of JSON.)
As before, the config system replaces the value node in the
containing config tree with the config tree resulting from the additional parse.
== Config Keys with . in name
As described in the xref:hierarchical-features.adoc#accessByKey[hierarchical features
section] each config node (except the root) has a non-null key.
[IMPORTANT]
=========
To emphasize, the dot character ("`.`") has special meaning as a name separator
in keys. To include a dot as a character in a key escape it as
"`~1`".
=========
For example, the following configuration file contains two object nodes with
names `oracle` and `oracle.com`.
[source,json]
.Example `application.json` with dot character in key
----
{
"oracle" : {
"com" : true,
"cz" : false
},
"oracle.com" : {
"secured" : true
}
}
----
[source,java]
.Working with configuration with dot character in node name
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_13, indent=0]
----
<1> Work with the first `oracle` object as usual.
As always you can use the fully-qualified key `oracle.com` or chain `get(key)`
calls to access the `com` property value.
<2> Config node `"oracle"` / `"com"` is a leaf node (has type `VALUE`)...
<3> ... and has the name `com` (the last token in its key).
<4> The second object has name `oracle.com`. The code must escape the
dot in the node's name using `oracle~1com`.
<5> Or, use the utility method `Config.Key.escapeName(name)` to escape dots or tildes
that might be in the node's name, in this example in `oracle.com`.
<6> The config node `"oracle.com"` has type `OBJECT`...
<7> ...and name `"oracle.com"`.
== Filters, Overrides, and Token Substitution [[filters-and-overrides]]
When your application retrieves a config value, the config system can transform it
before returning the value, according to _filters_, _overrides_, and
_tokens_. The config system provides some built-in instances of these
you can use, and you can add your own as described in the
sections which describe
xref:extensions.adoc#Config-SPI-ConfigFilter[filters] and
xref:extensions.adoc#Config-SPI-OverrideSource[overrides].
Your application can add filters and overrides explicitly to a config builder
and the config system by default uses the Java service loader mechanism to
locate all available filters and overrides and add them automatically to all
config builders (unless your code disables that behavior for a given
builder).
=== Filters
Each filter accepts a key and the value as defined in the source, and returns
the value to be used. The filter can leave the value unchanged or
alter it, as it sees fit.
The built-in link:{config-javadoc-base-url}/io/helidon/config/ConfigFilters.html#valueResolving--[value-resolving]
filter enables the token substitution described below.
See the link:{config-javadoc-base-url}/io/helidon/config/spi/ConfigFilter.html[`ConfigFilter`] JavaDoc
for more information.
=== Overrides
The overrides feature allows you to create an external document containing key/value
pairs which replace the value otherwise returned for the name, and then add that
document as an override source to a config builder.
There are some key differences between overrides and filters.
* Because overrides are loaded
from sources those sources can change while your application runs and so the
overrides they that prescribe can change.
* The override document can use wildcards in key expressions.
* Overrides can affect only keys that already exist in the original source; filters
can supply values even if the key is absent from the config source.
Each override entry consists of a Java properties-format definition. The key is an
expression (which can use wildcards) to match config keys read from the current
config sources, and the override value is the new value for any key matching the
key expression from that entry. Order is important. The
config system tests every key expression/value pair one by one in the order they appear
in the overrides sources. Once the config system finds an override entry in which
the key expression matches the configuration key, the system returns that entry's
value for the key being processed.
See the link:{config-javadoc-base-url}/io/helidon/config/spi/OverrideSource.html[`OverrideSource]` JavaDoc
for more detail.
=== Tokens
A token reference is a key token starting with `$`, optionally enclosed between
`{` and `}`, i.e. `$ref`, `pass:[${ref}]`. Even a key composed of more than one token
can be referenced in another key, i.e. `${env.ref}`.
As an example use case, you can use token references to declare the default values (see
`resolving-tokens.yaml` below), while the references may be resolved in another
config source, which identifies a current environment (see `env.yaml` examples
below). You can then use the same overrides for different environments, say `test` and `prod`.
The configuration in each environment is then overridden with a different values
using wildcards (see `overrides.properties` below).
[source,java]
.Initialize `Config` with Override Definition from `overrides.properties` file
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_14, indent=0]
----
<1> Loads _overrides_ from the specified file.
<2> A deployment-specific environment configuration file.
<3> A default configuration containing token references that are resolved
using the environment-specific override.
You can disable key and value token replacement separately as the following example shows.
[source,java]
.Disabling Key and Value Token Replacement
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_15, indent=0]
----
== Executors for Asynchronous Config Activity
Various parts of the config system work asynchronously:
* polling strategies to detect changes to config sources,
* publishers to notify your application when such changes occur,
* `Config` instances which subscribe to and respond to change notifications for
their underlying sources, and
* retry policies (which might wait between retries).
Each of these uses an executor to perform its work. The config system provides default
executors, but your application can specify different ones if necessary.
=== Executors for Polling Strategy
The method `PollingStrategies.regular(Duration)` returns builder for polling strategy.
This builder provides `executor` method which your application can invoke, passing a
`java.util.concurrent.ScheduledExecutorService` instance it requires for the
polling work. By default, each polling strategy instance uses a separate thread
pool executor.
The following example shares the same executor for two different polling
strategy instances.
[source,java]
.Customize polling strategy executors
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_16, indent=0]
----
<1> Prepares a thread pool executor with core pool size set `2`.
<2> Selects the built-in periodic polling strategy.
<3> Tells the config system to use the specific executor to poll the
`dev.properties` config source.
<4> Tells the config system to use the specific executor to poll the
`config.properties` config source.
=== Executors for Source Change Events
Recall that when a change watcher detects a change in a source, it informs
interested parties of the changes. By default, each `Config.Builder` arranges
for the resulting `Config` tree to use a shared executor that reuses available threads
from a pool, creating new threads as needed. The same executor is used for actually
reloading the source.
Your application can invoke the system watcher builder's `executor` method to
tell the builder to use a different `Executor`.
[source,java]
.Customize config and override sources' executors
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_17, indent=0]
----
<1> Prepares a thread pool executor to be shared by selected sources.
<2> Tells the builder that the resulting overrides source should use the specified
`Executor` for notifying interested parties of changes and for reloading the
override source.
<3> Uses the same `Executor` and event buffer size for the config source as for
the override source above.
=== Retry Policy Custom Executor
You can control which executor a retry policy should use for its work.
The `RetryPolicies.repeat(int retries)` method returns
a link:{config-javadoc-base-url}/io/helidon/config/SimpleRetryPolicy.Builder.html[SimpleRetryPolicy.Builder].
Your application can invoke the retry policy builder's `executorService` method to
specify which `ScheduledExecutorService` instance it should use to schedule and
execute delayed retries. By default, the config system uses a separate thread
pool executor for each retry policy instance.
[source,java]
.Customize retry policy executors
----
include::{sourcedir}/se/config/AdvancedConfigurationSnippets.java[tag=snippet_18, indent=0]
----
<1> Prepares a thread pool executor with core pool size set to `2` and a custom
`java.util.concurrent.ThreadFactory`.
<2> When the source is flagged as `optional()`, the loading attempt will be
repeated as the retry policy defines, but an overall failure will _not_ lead to
failing the initial load or preventing the source from being polled if so configured.
<3> Uses the built-in _repeating_ implementation of `RetryPolicy` that can be used with any
config source, but typically for ones that might suffer brief, intermittent outages.
<4> Specifies the executor to use for loading and retries.