org.apache.spark.sql.jdbc.README.md Maven / Gradle / Ivy
---
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
# JDBC Connection Provider Handling In Spark
This document aims to explain and demystify JDBC connection providers as they are used by Spark
and the usage or custom provider implementation is not obvious.
## What are JDBC connection providers and why use them?
Spark initially provided non-authenticated or user/password authenticated connections.
This is quite insecure and must be avoided when possible.
JDBC connection providers (CPs from now on) are making JDBC connections initiated by JDBC sources
which can be a more secure alternative.
Spark provides two ways to deal with stronger authentication:
* Built-in CPs added which support kerberos authentication using `keytab` and `principal` (but only if the JDBC driver supports keytab)
* `org.apache.spark.sql.jdbc.JdbcConnectionProvider` developer API added which allows developers
to implement any kind of database/use-case specific authentication method.
## How JDBC connection providers loaded?
CPs are loaded with service loader independently. So, if one CP is failed to load it has no
effect on all other CPs.
## How to disable JDBC connection providers?
There are cases where the built-in CP doesn't provide the exact feature which needed
so they can be turned off and can be replaced with custom implementation. All CPs must provide a `name`
which must be unique. One can set the following configuration entry in `SparkConf` to turn off CPs:
`spark.sql.sources.disabledJdbcConnProviderList=name1,name2`.
## How to enforce a specific JDBC connection provider?
When more than one JDBC connection provider can handle a specific driver and options, it is possible to
disambiguate and enforce a particular CP for the JDBC data source. One can set the DataFrame
option `connectionProvider` to specify the name of the CP they want to use.
## How a JDBC connection provider found when new connection initiated?
When a Spark source initiates JDBC connection it looks for a CP which supports the included driver,
the user just need to provide the `keytab` location and the `principal`. The `keytab` file must exist
on each node where connection is initiated.
CPs has a mandatory API which must be implemented:
`def canHandle(driver: Driver, options: Map[String, String]): Boolean`
If this function returns `true` then `Spark` considers the CP can handle the connection setup.
Built-in CPs returning `true` in the following cases:
* If the connection is not secure (no `keytab` or `principal` provided) then the `basic` named CP responds.
* If the connection is secure (`keytab` and `principal` provided) then the database specific CP responds.
Database specific providers are checking the JDBC driver class name and the decision is made based
on that. For example `PostgresConnectionProvider` responds only when the driver class name is `org.postgresql.Driver`.
Important to mention that exactly one CP must return `true` from `canHandle` for a particular connection
request because otherwise `Spark` can't decide which CP need to be used to make the connection.
Such cases exception is thrown and the data processing stops.
## How to implement a custom JDBC connection provider?
Spark provides an example CP in the examples project (which does nothing).
There are basically 2 files:
* `examples/src/main/scala/org/apache/spark/examples/sql/jdbc/ExampleJdbcConnectionProvider.scala`
which contains the main logic that can be further developed.
* `examples/src/main/resources/META-INF/services/org.apache.spark.sql.jdbc.JdbcConnectionProvider`
which manifest file is used by service loader (this tells `Spark` that this CP needs to be loaded).
Implementation considerations:
* CPs are running in heavy multi-threaded environment and adding a state into a CP is not advised.
If any state added then it must be synchronized properly. It could cause quite some headache to
hunt down such issues.
* Some of the CPs are modifying the JVM global security context so if the CP's
`modifiesSecurityContext` method returns `true` then the CP's `getConnection` method will
be called synchronized by `org.apache.spark.security.SecurityConfigurationLock` to avoid race.