x-infinispan.4.4.3.source-code.index.adoc Maven / Gradle / Ivy
= Infinispan Cluster Manager
This is a cluster manager implementation for Vert.x that uses https://infinispan.org/[Infinispan].
This implementation is packaged inside:
[source,xml,subs="+attributes"]
----
${maven.groupId}
${maven.artifactId}
${maven.version}
----
In Vert.x a cluster manager is used for various functions including:
* Discovery and group membership of Vert.x nodes in a cluster
* Maintaining cluster wide topic subscriber lists (so we know which nodes are interested in which event bus addresses)
* Distributed Map support
* Distributed Locks
* Distributed Counters
Cluster managers *do not* handle the event bus inter-node transport, this is done directly by Vert.x with TCP connections.
== Using this cluster manager
If you are using Vert.x from the command line, the jar corresponding to this cluster manager (it will be named `${maven.artifactId}-${maven.version}.jar`
should be in the `lib` directory of the Vert.x installation.
If you want clustering with this cluster manager in your Vert.x Maven or Gradle project then just add a dependency to
the artifact: `${maven.groupId}:${maven.artifactId}:${maven.version}` in your project.
If the jar is on your classpath as above then Vert.x will automatically detect this and use it as the cluster manager.
Please make sure you don't have any other cluster managers on your classpath or Vert.x might
choose the wrong one.
You can also specify the cluster manager programmatically if you are embedding Vert.x by specifying it on the options
when you are creating your Vert.x instance, for example:
[source,$lang]
----
{@link examples.Examples#createClusterManagerProgramatically()}
----
== Configuring this cluster manager
The default cluster manager configuration can be modified with `infinispan.xml` and/or `jgroups.xml` files.
The former configures the data grid, the latter group management and member discovery.
You can place one or both of them on your classpath.
If you want to embed your custom file in a fat jar, it must be located at the root of the fat jar.
If it's an external file, the **directory** containing the file must be added to the classpath. For
example, if you are using the _launcher_ class from Vert.x, the classpath enhancement can be done as follows:
[source,shell]
----
# If infinispan.xml and/or jgroups.xml files are in the current directory:
java -jar my-app.jar -cp . -cluster
# If infinispan.xml and/or jgroups.xml files are in the conf directory:
java -jar my-app.jar -cp conf -cluster
----
Another way to override the configuration is by providing the file locations via system properties:
`vertx.infinispan.config` and/or `vertx.jgroups.config`.
[source,shell]
----
# Use a cluster configuration located in an external file
java -Dvertx.infinispan.config=./config/my-infinispan.xml -jar ... -cluster
# Or use a custom configuration from the classpath
java -Dvertx.infinispan.config=my/package/config/my-infinispan.xml -jar ... -cluster
----
The cluster manager will search for the file in classpath first, and fallback to the filesystem.
The system properties, when present, override any `infinispan.xml` or `jgroups.xml` on the classpath.
The xml files are Infinispan and JGroups configuration files and are described in detail in the documentation on the Infinispan and JGroups web-sites.
IMPORTANT: if a `jgroups.xml` file is on the classpath or if you set the `vertx.jgroups.config` system property,
it will override any JGroups `stack-file` path defined in the Infinispan configuration file.
The default JGroups configuration uses multicast for discovery and TCP for group management.
Make sure multicast is enabled on your network for this to work.
For full documentation on how to configure the transport differently or use a different transport please consult the
Infinispan / JGroups documentations.
== Using an existing Infinispan Cache Manager
You can pass an existing `DefaultCacheManager` in the cluster manager to reuse an existing cache manager:
[source,$lang]
----
{@link examples.Examples#useExistingCacheManager(org.infinispan.manager.DefaultCacheManager)}
----
In this case, vert.x is not the cache manager owner and so do not shut it down on close.
Notice that the custom Infinispan instance need to be configured with:
[source,xml]
----
----
== Packaging an executable uber JAR
Infinispan uses Java's `ServiceLoader` mechanism to discover implementations of a few classes at runtime.
You must configure your build tool to merge service descriptors files when creating an executable uber JAR (also known as "fat" JAR).
If you use Maven and the Maven Shade Plugin, the plugin configuration should look like:
[source,xml]
----
----
If you use Gradle and the Gradle Shadow Plugin:
[source,kotlin]
----
shadowJar {
mergeServiceFiles()
}
----
Besides, Infinispan relies on https://openjdk.org/jeps/238[Multi-Release JAR Files] to allow multiple, Java-release-specific versions of some classes.
Consequently, the executable uber JAR's manifest file must contain the following entry:
----
Multi-Release: true
----
== Configuring for Kubernetes
On Kubernetes, JGroups can be configured to use either the Kubernetes API (`KUBE_PING`) or DNS (`DNS_PING`) for discovery.
In this document, we will use DNS discovery.
First, force usage of IPv4 in the JVM with a system property.
[source,shell]
----
-Djava.net.preferIPv4Stack=true
----
Then, set the `vertx.jgroups.config` system property to `default-configs/default-jgroups-kubernetes.xml`.
This JGroups stack file is located in the `infinispan-core` JAR and preconfigured for Kubernetes.
[source,shell]
----
-Dvertx.jgroups.config=default-configs/default-jgroups-kubernetes.xml
----
Also, set the JGroups DNS query to find members.
[source,shell]
----
-Djgroups.dns.query=MY-SERVICE-DNS-NAME
----
The `MY-SERVICE-DNS-NAME` value must be a https://kubernetes.io/docs/user-guide/services/#headless-services[*headless* Kubernetes service] name that will be used by JGroups to identify all cluster members.
A headless service can be created with:
[source,yaml]
----
apiVersion: v1
kind: Service
metadata:
name: clustered-app
spec:
selector:
cluster: clustered-app # <2>
ports:
- name: jgroups
port: 7800 # <1>
protocol: TCP
publishNotReadyAddresses: true # <3>
clusterIP: None
----
<1> JGroups TCP port
<2> Cluster members selected by the `cluster=clustered-app` label
<3> Set to true so that members can be discovered without interfering with your readiness probe logic
Eventually, apply the `cluster=clustered-app` label to all deployments that should be part of the cluster:
[source,yaml]
----
apiVersion: apps/v1
kind: Deployment
spec:
template:
metadata:
labels:
cluster: clustered-app
----
=== Rolling updates
During rolling udpates, the Infinispan team http://infinispan.org/docs/stable/user_guide/user_guide.html#using_kubernetes_and_openshift_rolling_updates[recommends] to replace pods one by one.
To do so, we must configure Kubernetes to:
* never start more than one new pod at once
* forbid more than one unavailable pod during the process
[source,yaml]
----
spec:
strategy:
type: Rolling
rollingParams:
updatePeriodSeconds: 10
intervalSeconds: 20
timeoutSeconds: 600
maxUnavailable: 1 <1>
maxSurge: 1 <2>
----
<1> the maximum number of pods that can be unavailable during the update process
<2> the maximum number of pods that can be created over the desired number of pods
Also, pod readiness probe must take cluster health into account.
Please refer to the <> section for details on how to implement a readiness probe with link:../../vertx-health-check/$lang/[Vert.x Health Checks].
== Configuring for Docker Compose
Make sure to start the Java Virtual Machines with those system properties:
[source,shell]
----
-Djava.net.preferIPv4Stack=true -Djgroups.bind.address=NON_LOOPBACK
----
This will make JGroups pick the interface of the virtual private network created by Docker.
== Trouble shooting clustering
If the default multicast discovery configuration is not working here are some common causes:
=== Multicast not enabled on the machine.
It is quite common in particular on OSX machines for multicast to be disabled by default. Please google for
information on how to enable this.
=== Using wrong network interface
If you have more than one network interface on your machine (and this can also be the case if you are running
VPN software on your machine), then JGroups may be using the wrong one.
To tell JGroups to use a specific interface you can provide the IP address of the interface in the `bind_addr`
element of the configuration. For example:
[source,xml]
----
© 2015 - 2025 Weber Informatics LLC | Privacy Policy