Skip to content

[Connectors] Add support for user-defined functions for pre-parsing in connectors #5608

@mihaibudiu

Description

@mihaibudiu

There should be a trait for describing a "decoder" for data in a connector. A decoder comprises

  • an initialization function, called once on pipeline startup
  • a decoder(&[u8]) -> Result<Vec<u8>, Error> function

The configuration for a connector will specify the name of an encoder. The user has to implement this trait for the encoder in the udf.rs file. The decoder function will be hooked to be executed between transport and parsing, and will pass the result to the parser for further processing. The design could potentially allow for a stack of such decoders for each connector.

Metadata

Metadata

Assignees

No one assigned

    Labels

    connectorsIssues related to the adapters/connectors cratehigh priorityTask should be tackled first, added in the current sprint if necessary

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions