Export Service Overview

The Export Services Layer provides a set of microservices to perform the following activities:

  • Allowing off-gateway clients to register for relevant data from the South Side objects
  • Defining where and when to deliver the data
  • Defining the format and shape in which to deliver the data

The information required for Export Service registration is described below.

Export Service Configuration

To configure an Export Service, you must define the MaxRetentionEvents, as described below.

MaxRetentionEvents

Defines the number of events that can be stored by each registration during the batch-export process, if enabled.

The value can be updated using Consul or a cURL command. The required setting depends on the following:

  • Amount of data to be stored during the batch-export process
  • Available memory on the machine running Edge Xpert

This value takes priority over the maxRetentionEvents value in each registration. For further information on the maxRetentionEvents member, see Maximum Retention Events.

To make changes using Consul, complete the following steps:

  1. Open a browser
  2. Navigate to the Consul GUI, at http://localhost:8500/ui
  3. In the GUI, select Key/Value from the menu
  4. Select the edgex folder
  5. Select the core folder
  6. Select the 1.0 folder
  7. Select the edgex-export-distro folder
  8. Select the Writeable folder
  9. Select MaxRetentionEvents
  10. Edit the value in the GUI, as required
  11. Select the Save button

To make changes using a cURL command, send the following command replacing {VALID_VALUE} with the value to apply:

curl --request PUT --data {VALID_VALUE} http://localhost:8500/v1/kv/edgex/core/1.0/edgex-export-distro/Writable/MaxRetentionEvents

Value Type: UINT

Default: 20000

Valid Values: A valid unsigned integer representing the maximum number of events to store for each registration during the batch-export process

Export Service Registration

An Export Service is registered using a cURL command. The command is similar to the following, although the members differ depending on the Export Service being used:

{
  "name": "Reg1",
  "addressable": {
    "name": "Reg1",
    "protocol": "TLS",
    "address": "test.mosquitto.org",
    "port": 8884,
    "publisher": "EdgeXExportPublisher",
    "topic": "EdgeXEvents_Reg1",
    "cert": "/export/keys/client.crt",
    "key": "/export/keys/client.key"
  },
  "format": "JSON",
  "filter":{
    "deviceIdentifiers":[
      "Random-Boolean-Device", "Random-Integer-Device"
    ]
  },
  "enable": true,
  "destination": "MQTT_TOPIC",
  "processFrequency": "10m",
  "maxBatchEvents": 900,
  "maxRetentionEvents": 900,
  "needDataType": true
}

To register an Export Service, you must define a combination of the following:

Note

The members that must be defined for a specific Export Service differ. The required members for supported Export Services are given in the examples provided for each Export Service.

Registration Name

Defines the name of the export-client registration. Required.

Format: "name": "<name>",

Value Type: String

Default: N/A

Valid Values: Single string identifying the export-client registration

Encryption

Defines the encryption to use when sending data to the export-client, using a key and initializing vector. If encryption is not used, this member can be omitted.

Format: "encryption": {[<sub-members>]},

The Encryption member has the following sub-members:

Encryption Algorithm

Defines the algorithm to use if using encryption.

Format: "encryptionAlgorithm": "<algorithm>",

Value Type: String

Default: N/A

Valid Values: AES

Encryption Key

Defines the key to use if using encryption.

Format: "encryptionKey": "<key>",

Value Type: String

Default: N/A

Valid Values: Valid client-provided key

Initializing Vector

Defines the initializing vector to use if using encryption.

Format: "initializingVector": "<initializing vector>",

Value Type: String

Default: N/A

Valid Values: Arbitrary number

Compression

Defines the client-requested method of compression to use when delivering the data to the export-client if the export-client has requested that the data be compressed. If compression is not used, this member can be omitted.

Format: "compression": "<method>",

Value Type: String

Default: N/A

Valid Values: GZIP, ZIP

Addressable

Defines the information necessary to identify and connect to the endpoint. Required.

Format: "addressable": {[<sub-members>]},

The Addressable member has the following sub-members:

Name

Defines the name of the endpoint.

Format: "name": "<name>",

Value Type: String

Default: N/A

Valid Values: Single string identifying the endpoint

Protocol

Defines the protocol used for communication.

Format: "protocol": "<protocol>",

Value Type: String

Default: N/A

Valid Values: String value identifying the protocol

Address

Defines the URI of the endpoint.

Format: "address": "<URI>"

Value Type: String

Default: N/A

Valid Values: Address of the API endpoint, often an URL such as xxx.yyy.com or an IP address

Port

Defines the port to use to connect to the endpoint.

Format: "port": <port number>,

Value Type: UINT

Default: N/A

Valid Values: The number of an available, valid port number

Publisher

Defines the MQTT Client ID to use. This applies only to MQTT export-client addressables.

Format: "publisher": "<MQTT Client ID>",

Value Type: String

Default: N/A

Valid Values: A valid MQTT Client ID

User

Defines the log in credentials for the Cloud or remote server.

Format: "user:" "<username>",

Value Type: String

Default: N/A

Valid Values: A valid username, or unused

For examples of use, see Setting Up the Google Cloud IoT Core Export Service and Setting Up the Microsoft Azure IoT Hub Export Service

Password

Defines the password for the Cloud or remote server. If a password is not used, this member can be omitted.

Format: "password:" "<password>",

Value Type: String

Default: N/A

Valid Values: The password or the name of the generated JWT.IO file

For examples of use, see Setting Up the Google Cloud IoT Core Export Service and Setting Up the Microsoft Azure IoT Hub Export Service

Method

Defines the HTTP request method to use. If HTTP requests are not used, this member can be omitted.

Format: "method:" "<method>",

Value Type: String

Default: N/A

Valid Values: POST

Topic

Defines the topic to publish.

Format: "topic:" "<topic>",

Value Type: String

Default: N/A

Valid Values: A valid topic, or blank

Certificate

Defines the certificate to use for this export-client. Different certificates can be used for each export-client, allowing the Export Service to export data to multiple destinations. Paired with the key member, described below. If certificates are not used, this member can be omitted.

Note

  1. The path to the certificate is inside the container.
  2. If using MQTT with secure (TLS) communication, you only require the certificate file for encryption. This means that the key does not need to be specified.

Format: "cert:" "<path/cert>",

Value Type: String

Default: If using TLS authentication but no certificate is specified, defaults to the setting in the configuration.toml file

Valid Values: A valid path and filename identifying the certificate

Key

Defines the key to use for this export-client. Different keys can be used for each export-client, allowing the Export Service to export data to multiple destinations. Paired with the cert member, described above. If keys are not used, this member can be omitted

Note

  1. The path to the key is inside the container.
  2. If using MQTT with secure (TLS) communication, you only require the certificate file for encryption. This means that the key does not need to be specified.

Format: "key:" "<path/key>"

Value Type: String

Default: If using TLS authentication but no key is specified, defaults to the setting in the configuration.toml file

Valid Values: A valid path and filename identifying the key

Format

Defines the format for the exported data. Required.

Format: "format:" "<format>"

Value Type: String

Default: N/A

Valid Values: AWS_JSON, AZURE_JSON, IOTCORE_JSON, INFLUXDB, JSON, NOOP, SERIALIZED, THINGSBOARD_JSON, XML

Filter

Defines the filter applied to the data to identify the data to export. If all data is to be exported, this member can be omitted.

Format: "filter:" "{"deviceIdentifiers":["<DeviceName>"] "resourceIdentifiers":["<ResourceName">]},

Value Type: String

Default: N/A

Valid Values: A comma-separated list of valid device names and/or a comma-separated list of valid resource names

Enable

Defines whether the Export Service is enabled or not. Required.

Format: "enable: "<true|false>",

Value Type: Bool

Default: true

Valid Values: To enable, set to true, to disable set to false

Destination

Defines the endpoint type. Required.

Format: "destination": "<endpoint type>",

Value Type: String

Default: N/A

Valid Values: AWS_TOPIC, AZURE_TOPIC, INFLUX_ENDPOINT, IOCORE_TOPIC, MQTT_TOPIC, REST_ENDPOINT, XMPP_TOPIC, ZMQ_TOPIC

Process Frequency

Defines the schedule for processing events using ISO 8601 format times. Batch processing can be enabled by setting the value to a specific frequency; that is, not blank or 0. If omitted, the default value is used.

If batch processing is enabled, the following apply:

  • Each export-client has its own List and scheduled thread

  • When an event is received, the event ID is stored in a Redis List owned by the export-client. The list is in Redis database x, where x is the number of the Redis database for the Export-Distro service as defined in the [Databases] section of the configuration.toml file. The database contains the EventIdList-<registrationName>" key

    Note

    Redis database 0 is used by other Edge Xpert services. IOTech recommends that Redis database 0 is not used when receiving events from the export-client.

If batch processing is enabled and the scheduled thread is run according to the processFrequency, the flow is as follows:

  1. Identify all eventIDs form the List
  2. Query the event from core-data by ID
  3. Export the events retrieved from core-data
  4. If the export of any event fails:
    • Store the ID in the LIST
    • Process the ID again in the next batch process
    • If the limit defined by maxRetentionEvents is reached, the system drops the ID of the event that failed to export

Format: "processFrequency": "<frequency>",

Value Type: String

Default: 0 all events exported immediately, batch process disabled

Valid Values: Empty (uses default), 0, ISO 8601 format times

Maximum Batch Events

Defines the number of events to process in each batch. If omitted, the default value is used.

Format: "maxBatchEvents":<number of events>,

Value Type: UINT

Default: 0 export all events

Valid Values: Empty (uses default), 0, the number of events to process in each batch

Maximum Retention Events

Defines the number of events that can be stored by the registration. If omitted, the default value is used.

If the defined number of events is reached, new events are stored and old data removed on a First-in, First-out basis.

If the incoming number of events exceeds the defined limit, only the last events are retained and all older events are discarded.

Format: "maxRetentionEvents": <number to retain>,

Value Type: UINT

Default: 20000

Valid Values: A valid integer representing the maximum number of events that can be retained by the registration

Need Data Type

Defines whether to add data type information to the readings. Required.

Format: "needDataType": <true|false>

Value Type: BOOL

Default: false

Valid Values: To add data type information, set to true, to omit data type information set to false