org.apache.beam.examples.twitterstreamgenerator.README.md Maven / Gradle / Ivy
Go to download
Show more of this group Show more artifacts with this name
Show all versions of beam-examples-java-twitter Show documentation
Show all versions of beam-examples-java-twitter Show documentation
Apache Beam SDK provides a simple, Java-based
interface for processing virtually any size data. This
artifact includes all Apache Beam Java SDK examples.
# Twitter Connector
This directory contains an example pipelines for how to perform continues stream of data from twitter streaming api ( or any other 3rd party API ). This include:
- Splitable Dofn
— A simple example of implementation of splittable dofn on an unbounded source with a simple incrementing watermarking logic.
- Connection Management
— The streaming pipeline example makes sure that only one Twitter connection is active at a time for a configuration.
- Terminating pipeline by time or elements
— The streaming pipeline keeps track of time and data collecting so far and terminated when the limit specified in passed.
## Requirements
- Java 8
- Twitter developer app account and streaming credentials.
- Direct runner or Flink runner.
This section describes what is needed to get the example up and running.
- Gradle preparation
- Local execution