Package org.apache.druid.catalog.model.table

Class HttpInputSourceDefn

java.lang.Object
org.apache.druid.catalog.model.table.BaseInputSourceDefn
org.apache.druid.catalog.model.table.FormattedInputSourceDefn
org.apache.druid.catalog.model.table.HttpInputSourceDefn
All Implemented Interfaces:
InputSourceDefn
public class HttpInputSourceDefn extends FormattedInputSourceDefn
Definition of an HTTP input source source.

Provides a parameterized form where the user defines a value for the uriTemplate table property in the table spec, then provides the partial URLs in a table function to use for that one query. The final URIs are created by combining the template and the arguments. Example:

  • uriTemplate property): "http://example.com/data/kttm-{}.json"
  • uris function argument: "22-Nov-21, 22-Nov-22"

    • When the template is used, the format is optional: it can be provided either with the table spec or at runtime, depending on what the user wants to accomplish. In the above, where the ".json" is encoded in the template, it makes sense to include the format with the spec. If the template was "http://example.com/data/{}", and the data comes in multiple formats, it might make sense to specify the format in the query. In this case, the table spec acts more like a connection.

    If the template is not used, then the uris property must be provided in the table spec, along with the corresponding format.

    The above semantics make a bit more sense when we realize that the spec can also provide a user name and password. When those are provided, then the input source must name a single site: the one for which the credentials are valid. Given this, the only table spec that makes sense is one where the URI is defined: either as a template or explicitly.

    When used as an ad-hoc function, the user specifies the uris and optional user name and password: the template is not available (or useful) in the ad-hoc case.

    Table function parameters are cleaned up relative to the input source field names to make them a bit easier to use.

    • Field Details

    • Constructor Details

      • HttpInputSourceDefn

        public HttpInputSourceDefn()

    • Method Details

      • typeValue

        public String typeValue()
        Description copied from interface: InputSourceDefn
        Type value for this format: same as the type string used in the serialized JSON for this input source. Used as the key for this definition within the table registry, and associates the serialized JSON with the corresponding input source definition.

      • inputSourceClass

        protected Class<? extends InputSource> inputSourceClass()
        Description copied from class: BaseInputSourceDefn
        Overridden by each subclass to return the input source class to be used for JSON conversions.
        Specified by:
        inputSourceClass in class BaseInputSourceDefn

      • validate

        public void validate(ResolvedExternalTable table)
        Description copied from interface: InputSourceDefn
        Given a external table catalog spec, with the JSON input source and format properties parsed to generic Java maps, validate that the properties are valid prior to saving the spec into the catalog.
        Specified by:
        validate in interface InputSourceDefn
        Overrides:
        validate in class FormattedInputSourceDefn
        Parameters:
        table - a catalog table spec with the input source and input format properties parsed into generic Java maps

      • auditInputSource

        protected void auditInputSource(Map<String,Object> jsonMap)
        Description copied from class: BaseInputSourceDefn
        Optional step to audit or adjust the input source properties prior to conversion via Jackson. Changes are made directly in the jsonMap.
        Overrides:
        auditInputSource in class BaseInputSourceDefn

      • adHocTableFnParameters

        protected List<TableFunction.ParameterDefn> adHocTableFnParameters()
        Description copied from class: FormattedInputSourceDefn
        Overridden by subclasses to provide the list of table function parameters for this specific input format. This list is combined with parameters for input formats. The method is called only once per run.
        Specified by:
        adHocTableFnParameters in class FormattedInputSourceDefn

      • convertArgsToSourceMap

        protected void convertArgsToSourceMap(Map<String,Object> jsonMap, Map<String,Object> args)
        Description copied from class: BaseInputSourceDefn
        Convert SQL arguments to the corresponding "generic JSON" form in the given map. The map will then be adjusted and converted to the actual input source.
        Specified by:
        convertArgsToSourceMap in class BaseInputSourceDefn

      • partialTableFn

        public TableFunction partialTableFn(ResolvedExternalTable table)
        Description copied from interface: InputSourceDefn
        Provide a definition for a SQL table function that completes a partial table spec from the catalog. Used when the spec represents a "partial table" or a "connection". The function provides parameters needed to complete the table (typically the set of input files, objects, etc.) If the catalog table spec does not provide a format, and this input source requires a format, then the parameters also include parameters for all supported input formats, same as for InputSourceDefn.adHocTableFn().
        Parameters:
        table - a catalog table spec with the input source and input format properties parsed into generic Java maps
        Returns:
        a fully-defined external table to be handed off to the Calcite planner.

      • convertCompletedTable

        protected ExternalTableSpec convertCompletedTable(ResolvedExternalTable table, Map<String,Object> args, List<ColumnSpec> columns)
        Description copied from class: BaseInputSourceDefn
        Complete a partial table using the table function arguments and columns provided. The arguments match the set of parameters used for the function. The columns are provided if the SQL included an EXTENDS clause: the implementation should decide if columns are required (or allowed) depending on whether the partial spec already defines columns.
        Specified by:
        convertCompletedTable in class BaseInputSourceDefn
        Parameters:
        table - the partial table spec, with input source and format parsed into a generic Java map
        args - the argument values provided in the SQL table function call. The arguments use the Java types defined in the parameter definitions.
        columns - the set of columns (if any) from the SQL EXTEND clause
        Returns:
        an external table spec which Calcite can consume