io.nosqlbench.engine.extensions.s3uploader.s3.md Maven / Gradle / Ivy

Go to download
Show more of this group Show more artifacts with this name
Show all versions of engine-extensions Show documentation
Scripting extensions for nosqlbench.
There is a newer version: 5.17.0
S3 extension
==============

Allow uploading of a local directory on the default filesystem
to an S3 bucket, using an S3 URI to specify the bucket, location, and so on.

The URL is specified in the standard S3 format, such as:

1. `s3://mybucket/mypath-as-a-key/with-any-level-of-depth`
2. `s3://myuser:mypass@mybucket/mypath-as-a-key/with-any-level-of-depth`

In addition, any tokens which are supported by the standard NoSQLBench
token substitution mechanism will be used to construct a URL at the time
of usage. These forms include the following:

- Scenario Metadata - There are several key fields initialized for a scenario which can be used as common
  reference points. These occlude the environment variables of the same name. These are:
  - SESSION_NAME - The name auto-generated for a session, used in the logfile names, and so on.
  - SYSTEM_ID - The string form of the most canonically identifying IP address, excluding
    known symbolic interface names (docker*, for example) and all localhost addresses.
  - SYSTEM_FINGERPRINT - a stable and anonymized identifier for a given system. This will be
    stable as long as the networking configuration does not change.
- System Properties
  - Any parameter in `$word1.word2...` form -- any multi-part variable name with separating dots
    is taken as a system property to the JVM. These are expanded in place. Both `$word1.word2`
    and `${word1.word2}` patterns are supported, whereas the latter is more strict and thus safer.
- Environment Variables
  - As with System Properties, environment variable form the shell are also supported, as long
    as they do not include a dot.
- Temporal Fields from the Scenario start time
  - Any field specifier that you can use with the temporal types in Java's standard String.
    format can be used. The reference time for these is always the scenario start time.
  - Example: The default session name template looks like `scenario_%tY%tm%td_%tH%tM%tS_%tL`

## Examples
```
// If you have local logical identifiers in your scenario script which you want
// to templatize into your upload paths, you can provide your own js object
// as the third parameter
s3.uploadDirToUrlTokenized(
 'metrics',
 's3://test-results/${HOSTNAME}/${testid}-${testversion}/metrics',
 {
  'testid':'20210343',
  'testversion':'v2'
 }
);

// Otherwise, use the two-parameter version:
s3.uploadDirToUrl('metrics','s3://test-results/${HOSTNAME}/metrics');
```

## Post-Hoc upload

Scripting extensions only run if the scenario is not halted before they are invoked
in the main scenario script. If you want to ensure that this one runs after a test,
regardless of when or why the test stopped, it is possible to wrap it within
a shutdown hook which will run after scenario completion.

This is an example of how to do so:

```
shutdown.addShutdownHook('upload_metrics', function f() {
 s3.uploadDirToUrl('metrics','s3://test-results/${HOSTNAME}/metrics');
});

```