Packages

case class MergedParameters(parameters: Map[String, String]) extends Product with Serializable

Adds validators and accessors to string map

Linear Supertypes
Serializable, Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MergedParameters
  2. Serializable
  3. Product
  4. Equals
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new MergedParameters(parameters: Map[String, String])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def autoPushdown: Boolean

    Snowflake automatically enable/disable pushdown function

  6. def awsAccessKey: Option[String]
  7. def awsSecretKey: Option[String]
  8. def azureSAS: Option[String]
  9. def bindVariableEnabled: Boolean
  10. def checkBucketConfiguration: Boolean

    Returns true if bucket lifecycle configuration should be checked

  11. def checkTableExistenceInCurrentSchemaOnly: Boolean
  12. def checkTableExistenceWithFullyQualifiedName: Boolean
  13. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native() @HotSpotIntrinsicCandidate()
  14. def columnMap: Option[Map[String, String]]

    Retrieve Column mapping data.

    Retrieve Column mapping data. None if empty

  15. def columnMapping: String
  16. def columnMismatchBehavior: String
  17. def continueOnError: Boolean

    Set on_error parameter to continue in COPY command todo: create data validation function in spark side instead of using COPY COMMAND

  18. def createPerQueryTempDir(): String

    Creates a per-query subdirectory in the rootTempDir, with a random UUID.

  19. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. def expectedPartitionCount: Int
  21. def expectedPartitionSize: Long
  22. def extraCopyOptions: String

    Extra options to append to the Snowflake COPY command (e.g.

    Extra options to append to the Snowflake COPY command (e.g. "MAXERROR 100").

  23. def forceSkipPrePostActionsCheck: Boolean
  24. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  25. def getQueryIDUrl(queryID: String): String
  26. def getQueryResultFormat: Option[String]

    Snowflake query result format

  27. def getS3StageVpceDnsName: Option[String]
  28. def getStringTimestampFormat: Option[String]
  29. def getTimeOutputFormat: Option[String]

    Snowflake time output format

  30. def isExecuteQueryWithSyncMode: Boolean
  31. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  32. def isSslON: Boolean
  33. def isTimestampSnowflake(timestampFormat: String): Boolean
  34. def isTimezoneSnowflake: Boolean
  35. def isTimezoneSnowflakeDefault: Boolean
  36. def isTimezoneSpark: Boolean
  37. def keepOriginalColumnNameCase: Boolean
  38. def maxRetryCount: Int
  39. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  40. def nonProxyHosts: Option[String]
  41. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  42. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  43. lazy val parallelism: Option[Int]

    Number of threads used for PUT/GET.

  44. val parameters: Map[String, String]
  45. def postActions: Array[String]

    List of semi-colon separated SQL statements to run after successful write operations.

    List of semi-colon separated SQL statements to run after successful write operations. This can be useful for running GRANT operations to make your new tables readable to other users and groups.

    If the action string contains %s, the table name will be substituted in, in case a staging table is being used.

    Defaults to empty.

  46. def preActions: Array[String]

    List of semi-colon separated SQL statements to run before write operations.

    List of semi-colon separated SQL statements to run before write operations. This can be useful for running DELETE operations to clean up data

    If the action string contains %s, the table name will be substituted in, in case a staging table is being used.

    Defaults to empty.

  47. def privateKey: Option[PrivateKey]

    Generate private key form pem key value

    Generate private key form pem key value

    returns

    private key object

  48. def productElementNames: Iterator[String]
    Definition Classes
    Product
  49. def productIterator: Iterator[Any]
    Definition Classes
    Product
  50. def proxyHost: Option[String]
  51. lazy val proxyInfo: Option[ProxyInfo]
  52. def proxyPassword: Option[String]
  53. def proxyPort: Option[String]
  54. def proxyProtocol: Option[String]
  55. def proxyUser: Option[String]
  56. def purge(): Boolean

    Whether or not to have PURGE in the COPY statement generated by the Spark connector

  57. def query: Option[String]

    The Snowflake query to be used as the target when loading data.

  58. def quoteJsonFieldName: Boolean
  59. lazy val rootTempDir: String

    A root directory to be used for intermediate data exchange, expected to be on cloud storage (S3 or Azure storage), or somewhere that can be written to and read from by Snowflake.

    A root directory to be used for intermediate data exchange, expected to be on cloud storage (S3 or Azure storage), or somewhere that can be written to and read from by Snowflake. Make sure that credentials are available for this cloud provider.

  60. lazy val rootTempDirStorageType: FSType
  61. def s3maxfilesize: String

    Max file size used to move data out from Snowflake

  62. def setColumnMap(fromSchema: Option[StructType], toSchema: Option[StructType]): Unit

    set column map

  63. def sfAccount: Option[String]

    Snowflake account - optional

  64. def sfAuthenticator: Option[String]

    Mapping OAuth and authenticator values

  65. def sfCompress: Boolean

    Snowflake use compression on/off - "on" by default

  66. def sfDatabase: String

    Snowflake database name

  67. def sfExtraOptions: Map[String, AnyRef]

    Returns a map of options that are not known to the connector, and are passed verbosely to the JDBC driver

  68. def sfFullURL: String

    URL pointing to the snowflake database including protocol.

    URL pointing to the snowflake database including protocol. for example, https://host:port

  69. def sfPassword: String

    Snowflake password

  70. def sfRole: Option[String]

    Snowflake role - optional

  71. def sfSSL: String

    Snowflake SSL on/off - "on" by default

  72. def sfSchema: String

    Snowflake schema

  73. def sfTimestampLTZOutputFormat: Option[String]
  74. def sfTimestampNTZOutputFormat: Option[String]
  75. def sfTimestampTZOutputFormat: Option[String]
  76. def sfTimezone: Option[String]

    Snowflake timezone- optional

  77. def sfToken: Option[String]
  78. def sfURL: String

    URL pointing to the snowflake database, simply host:port

  79. def sfUser: String

    Snowflake user

  80. def sfWarehouse: Option[String]

    Snowflake warehouse

  81. def skipWriteWhenWritingEmptyDataFrame: Boolean
  82. def stagingTableNameRemoveQuotesOnly: Boolean
  83. def storagePath: Option[String]
  84. def streamingStage: Option[String]
  85. def supportAWSStageEndPoint: Boolean
  86. def supportShareConnection: Boolean
  87. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  88. def table: Option[TableName]

    The Snowflake table to be used as the target when loading or writing data.

  89. def temporaryAWSCredentials: Option[AWSCredentials]

    Temporary AWS credentials which are passed to Snowflake.

    Temporary AWS credentials which are passed to Snowflake. These only need to be supplied by the user when Hadoop is configured to authenticate to S3 via IAM roles assigned to EC2 instances.

  90. def temporaryAzureStorageCredentials: Option[StorageCredentialsSharedAccessSignature]

    SAS Token to be passed to Snowflake to access data in Azure storage.

    SAS Token to be passed to Snowflake to access data in Azure storage. We currently don't support full storage account key so this has to be provided if customer would like to load data through their storage account directly.

  91. def toString(): String
    Definition Classes
    MergedParameters → AnyRef → Any
  92. def treatDecimalAsLong: Boolean
  93. def trimSpace: Boolean

    Boolean that specifies whether to remove white space from String fields Defaults to false

  94. def truncateColumns(): Boolean

    Whether or not to have TRUNCATE_COLUMNS in the COPY statement generated by the Spark connector.

  95. def truncateTable: Boolean

    Truncate table when overwriting.

    Truncate table when overwriting. Keep the table schema

  96. def uploadChunkSize: Int
  97. def useAWSRegionURL: Boolean
  98. def useAwsMultiplePartsUpload: Boolean
  99. def useCopyUnload: Boolean
  100. def useExponentialBackoff: Boolean
  101. def useParseJsonForWrite: Boolean
  102. def useProxy: Boolean

    Proxy related parameters.

  103. def useStagingTable: Boolean

    When true, data is always loaded into a new temporary table when performing an overwrite.

    When true, data is always loaded into a new temporary table when performing an overwrite. This is to ensure that the whole load process succeeds before dropping any data from Snowflake, which can be useful if, in the event of failures, stale data is better than no data for your systems.

    Defaults to true.

  104. lazy val usingExternalStage: Boolean
  105. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  106. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  107. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped