All Downloads are FREE. Search and download functionalities are using the official Maven repository.

com.azure.messaging.eventhubs.package-info Maven / Gradle / Ivy

There is a newer version: 5.19.2
Show newest version
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

/**
 * 

Azure Event Hubs is a highly scalable * publish-subscribe service that can ingest millions of events per second and stream them to multiple consumers. This * lets you process and analyze the massive amounts of data produced by your connected devices and applications. Once * Event Hubs has collected the data, you can retrieve, transform, and store it by using any real-time analytics * provider or with batching/storage adapters.

* *

The Azure Event Hubs client library allows Java developers to interact with Azure Event Hubs. It provides a set * of clients that enable Java developers to publish events to and consume events from an Event Hub.

* *

Key Concepts

* *
    *
  • Event Hub producer: A source of telemetry data, diagnostics information, usage logs, or * other data, as part of an embedded device solution, a mobile device application, a game title running on a * console or other device, some client or server based business solution, or a website
  • * *
  • Event Hub consumer: Fetches events published to an Event Hub and processes it. Processing * may involve aggregation, complex computation, and filtering. Processing may also involve distribution or storage * of the information in a raw or transformed fashion. Event Hub consumers are often robust and high-scale platform * infrastructure parts with built-in analytics capabilities, like Azure Stream Analytics, Apache Spark, or Apache * Storm.
  • * *
  • Partition: An ordered sequence of events that is held in an Event Hub. Azure Event Hubs * provides message streaming through a partitioned consumer pattern in which each consumer only reads a specific * subset, or partition, of the message stream. As newer events arrive, they are added to the end of this sequence. * The number of partitions is specified at the time an Event Hub is created and cannot be changed.
  • * *
  • Consumer group: A view of an entire Event Hub. Consumer groups enable multiple consuming * applications to each have a separate view of the event stream, and to read the stream independently at their own * pace and from their own position. There can be at most 5 concurrent readers on a partition per consumer group; * however it is recommended that there is only one active consumer for a given partition and consumer group * pairing. Each active reader receives the events from its partition; if there are multiple readers on the same * partition, then they will receive duplicate events.
  • * *
  • Stream offset: The position of an event within an Event Hub partition. It is a * client-side cursor that specifies the point in the stream where the event is located. The offset of an * event can change as events expire from the stream.
  • * *
  • Stream sequence number: A number assigned to the event when it was enqueued in the * associated Event Hub partition. This is unique for every message received in the Event Hub partition.
  • * *
  • Checkpointing: A process by which readers mark or commit their position within a * partition event sequence. Checkpointing is the responsibility of the consumer and occurs on a per-partition * basis within a consumer group. This responsibility means that for each consumer group, each partition reader * must keep track of its current position in the event stream, and can inform the service when it considers the * data stream complete.
  • *
* *

Getting Started

* *

Service clients are the point of interaction for developers to use Azure Event Hubs. * {@link com.azure.messaging.eventhubs.EventHubProducerClient} and * {@link com.azure.messaging.eventhubs.EventHubProducerAsyncClient} are the sync and async clients for publishing * events to an Event Hub. Similarly, {@link com.azure.messaging.eventhubs.EventHubConsumerClient} and * {@link com.azure.messaging.eventhubs.EventHubConsumerAsyncClient} are the sync and async clients for consuming * events from an Event Hub. In production scenarios, we recommend users leverage * {@link com.azure.messaging.eventhubs.EventProcessorClient} because consumes events from all Event Hub partition, load * balances work between multiple instances of {@link com.azure.messaging.eventhubs.EventProcessorClient} and can * perform checkpointing.

* *

The examples shown in this document use a credential object named DefaultAzureCredential for authentication, * which is appropriate for most scenarios, including local development and production environments. Additionally, we * recommend using * managed identity * for authentication in production environments. You can find more information on different ways of authenticating and * their corresponding credential types in the * Azure Identity documentation". *

* *

Publishing events

* *

This library provides several ways to publish events to Azure Event Hubs. There is a producer client, that sends * events immediately to Azure Event Hubs and a buffered producer, that batches events together in the background and * publishes them later. These two clients have synchronous and asynchronous versions. The samples below demonstrate * simple scenarios, more snippets can be found in the class documentation for * {@link com.azure.messaging.eventhubs.EventHubProducerClient}, * {@link com.azure.messaging.eventhubs.EventHubProducerAsyncClient}, * {@link com.azure.messaging.eventhubs.EventHubBufferedProducerClient}, and * {@link com.azure.messaging.eventhubs.EventHubBufferedProducerAsyncClient}.

* *

In the following snippets, {@code fullyQualifiedNamespace} is the Event Hubs Namespace's host name. It is listed * under the "Essentials" panel after navigating to the Event Hubs Namespace via Azure Portal. The credential used is * {@code DefaultAzureCredential} because it combines commonly used credentials in deployment and development and * chooses the credential to used based on its running environment.

* *

Sample: Construct a synchronous producer and publish events

* *

The following code sample demonstrates the creation of the synchronous client * {@link com.azure.messaging.eventhubs.EventHubProducerClient}.

* * *
 * TokenCredential credential = new DefaultAzureCredentialBuilder().build();
 *
 * EventHubProducerClient producer = new EventHubClientBuilder()
 *     .credential("<<fully-qualified-namespace>>", "<<event-hub-name>>",
 *         credential)
 *     .buildProducerClient();
 *
 * List<EventData> allEvents = Arrays.asList(new EventData("Foo"), new EventData("Bar"));
 * EventDataBatch eventDataBatch = producer.createBatch();
 *
 * for (EventData eventData : allEvents) {
 *     if (!eventDataBatch.tryAdd(eventData)) {
 *         producer.send(eventDataBatch);
 *         eventDataBatch = producer.createBatch();
 *
 *         // Try to add that event that couldn't fit before.
 *         if (!eventDataBatch.tryAdd(eventData)) {
 *             throw new IllegalArgumentException("Event is too large for an empty batch. Max size: "
 *                 + eventDataBatch.getMaxSizeInBytes());
 *         }
 *     }
 * }
 *
 * // send the last batch of remaining events
 * if (eventDataBatch.getCount() > 0) {
 *     producer.send(eventDataBatch);
 * }
 *
 * // Clients are expected to be long-lived objects.
 * // Dispose of the producer to close any underlying resources when we are finished with it.
 * producer.close();
 * 
* * *

Sample: Creating an {@link com.azure.messaging.eventhubs.EventHubBufferedProducerClient} and enqueuing * events

* *

The following code sample demonstrates the creation of the synchronous client * {@link com.azure.messaging.eventhubs.EventHubBufferedProducerClient} as well as enqueueing events. The producer is * set to publish events every 60 seconds with a buffer size of 1500 events for each partition.

* * *
 * TokenCredential credential = new DefaultAzureCredentialBuilder().build();
 *
 * // "<<fully-qualified-namespace>>" will look similar to "{your-namespace}.servicebus.windows.net"
 * // "<<event-hub-name>>" will be the name of the Event Hub instance you created inside the Event Hubs namespace.
 * EventHubBufferedProducerClient client = new EventHubBufferedProducerClientBuilder()
 *     .credential("fully-qualified-namespace", "event-hub-name", credential)
 *     .onSendBatchSucceeded(succeededContext -> {
 *         System.out.println("Successfully published events to: " + succeededContext.getPartitionId());
 *     })
 *     .onSendBatchFailed(failedContext -> {
 *         System.out.printf("Failed to published events to %s. Error: %s%n",
 *             failedContext.getPartitionId(), failedContext.getThrowable());
 *     })
 *     .buildClient();
 *
 * List<EventData> events = Arrays.asList(new EventData("maple"), new EventData("aspen"),
 *     new EventData("oak"));
 *
 * // Enqueues the events to be published.
 * client.enqueueEvents(events);
 *
 * // Seconds later, enqueue another event.
 * client.enqueueEvent(new EventData("bonsai"));
 *
 * // Causes any buffered events to be flushed before closing underlying connection.
 * client.close();
 * 
* * *

Consuming events

* *

This library provides several ways to consume events from Azure Event Hubs. There are consumer clients, * {@link com.azure.messaging.eventhubs.EventHubConsumerClient} and * {@link com.azure.messaging.eventhubs.EventHubConsumerAsyncClient}, which fetches events from either a single * partition or all partitions in an Event Hub. For production, we recommend * {@link com.azure.messaging.eventhubs.EventProcessorClient} whose checkpoints are backed by a durable storage such as * Azure Blob Storage. The samples below demonstrate simple scenarios, more snippets can be found in the class * documentation for {@link com.azure.messaging.eventhubs.EventHubConsumerClient}, * {@link com.azure.messaging.eventhubs.EventHubConsumerAsyncClient}, and * {@link com.azure.messaging.eventhubs.EventProcessorClient}.

* *

In the following snippets, {@code fullyQualifiedNamespace} is the Event Hubs Namespace's host name. It is listed * under the "Essentials" panel after navigating to the Event Hubs Namespace via Azure Portal. The credential used is * {@code DefaultAzureCredential} because it combines commonly used credentials in deployment and development and * chooses the credential to used based on its running environment. The {@code consumerGroup} is found by navigating * to the Event Hub instance, and selecting "Consumer groups" under the "Entities" panel. The {@code consumerGroup} is * required for creating consumer clients.

* *

The credential used is {@code DefaultAzureCredential} because it combines * commonly used credentials in deployment and development and chooses the credential to used based on its running * environment.

* *

Sample: Construct a synchronous consumer and receive events

* *

The following code sample demonstrates the creation of the synchronous client * {@link com.azure.messaging.eventhubs.EventHubConsumerClient}. In addition, it receives the first 100 events that * were enqueued 12 hours ago. If there are less than 100 events, the ones fetched within {@code maxWaitTime} of 30 * seconds are returned.

* * *
 * TokenCredential credential = new DefaultAzureCredentialBuilder().build();
 *
 * // "<<fully-qualified-namespace>>" will look similar to "{your-namespace}.servicebus.windows.net"
 * // "<<event-hub-name>>" will be the name of the Event Hub instance you created inside the Event Hubs namespace.
 * EventHubConsumerClient consumer = new EventHubClientBuilder()
 *     .credential("<<fully-qualified-namespace>>", "<<event-hub-name>>",
 *         credential)
 *     .consumerGroup(EventHubClientBuilder.DEFAULT_CONSUMER_GROUP_NAME)
 *     .buildConsumerClient();
 *
 * Instant twelveHoursAgo = Instant.now().minus(Duration.ofHours(12));
 * EventPosition startingPosition = EventPosition.fromEnqueuedTime(twelveHoursAgo);
 * String partitionId = "0";
 *
 * // Reads events from partition '0' and returns the first 100 received or until the 30 seconds has elapsed.
 * IterableStream<PartitionEvent> events = consumer.receiveFromPartition(partitionId, 100,
 *     startingPosition, Duration.ofSeconds(30));
 *
 * Long lastSequenceNumber = -1L;
 * for (PartitionEvent partitionEvent : events) {
 *     // For each event, perform some sort of processing.
 *     System.out.print("Event received: " + partitionEvent.getData().getSequenceNumber());
 *     lastSequenceNumber = partitionEvent.getData().getSequenceNumber();
 * }
 *
 * // Figure out what the next EventPosition to receive from is based on last event we processed in the stream.
 * // If lastSequenceNumber is -1L, then we didn't see any events the first time we fetched events from the
 * // partition.
 * if (lastSequenceNumber != -1L) {
 *     EventPosition nextPosition = EventPosition.fromSequenceNumber(lastSequenceNumber, false);
 *
 *     // Gets the next set of events from partition '0' to consume and process.
 *     IterableStream<PartitionEvent> nextEvents = consumer.receiveFromPartition(partitionId, 100,
 *         nextPosition, Duration.ofSeconds(30));
 * }
 * 
* * *

Sample: Construct an {@link com.azure.messaging.eventhubs.EventProcessorClient}

* *

The following code sample demonstrates the creation of the processor client. The processor client is recommended * for production scenarios because it can load balance between multiple running instances, can perform checkpointing, * and reconnects on transient failures such as network outages. The sample below uses an in-memory * {@link com.azure.messaging.eventhubs.CheckpointStore} but * * azure-messaging-eventhubs-checkpointstore-blob provides a checkpoint store backed by Azure Blob Storage. *

* * *
 * TokenCredential credential = new DefaultAzureCredentialBuilder().build();
 *
 * // "<<fully-qualified-namespace>>" will look similar to "{your-namespace}.servicebus.windows.net"
 * // "<<event-hub-name>>" will be the name of the Event Hub instance you created inside the Event Hubs namespace.
 * EventProcessorClient eventProcessorClient = new EventProcessorClientBuilder()
 *     .consumerGroup(EventHubClientBuilder.DEFAULT_CONSUMER_GROUP_NAME)
 *     .credential("<<fully-qualified-namespace>>", "<<event-hub-name>>",
 *         credential)
 *     .processEvent(eventContext -> {
 *         System.out.printf("Partition id = %s and sequence number of event = %s%n",
 *             eventContext.getPartitionContext().getPartitionId(),
 *             eventContext.getEventData().getSequenceNumber());
 *     })
 *     .processError(errorContext -> {
 *         System.out.printf("Error occurred in partition processor for partition %s, %s%n",
 *             errorContext.getPartitionContext().getPartitionId(),
 *             errorContext.getThrowable());
 *     })
 *     .checkpointStore(new SampleCheckpointStore())
 *     .buildEventProcessorClient();
 *
 * eventProcessorClient.start();
 *
 * // Continue to perform other tasks while the processor is running in the background.
 * //
 * // Finally, stop the processor client when application is finished.
 * eventProcessorClient.stop();
 * 
* * * @see com.azure.messaging.eventhubs.EventHubClientBuilder * @see com.azure.messaging.eventhubs.EventProcessorClientBuilder * @see Azure Event Hubs */ package com.azure.messaging.eventhubs;




© 2015 - 2024 Weber Informatics LLC | Privacy Policy