Querying historical data

This section explains how to read data points from the system's time-series database.

Api Key Authorization

Required FieldsValues
Context typeDevice or App
Scopesdevice:read-data

As mentioned before, all the properties with retention policy historical are automatically stored in the system's time-series database. The property values can be queried anytime by POSTing a query object to property's data endpoint as shown above.

Query Object

AttributeDescription
startAbsoluteThe time in ISO 8601 format.
startRelativeContains value and unit sub attributes. The relative start time is the current date and time minus the specified value and unit. Possible unit values are “milliseconds”, “seconds”, “minutes”, “hours”, “days”, “weeks”, “months”, and “years”. For example, if the start time is 5 minutes, the query will return all matching data points for the last 5 minutes.
endAbsoluteThe time in ISO 8601 format. This must be later in time than the start time. If not specified, the end time is assumed to be the current date and time.
endRelativeThe relative end time is the current date and time minus the specified value and unit. Possible unit values are “milliseconds”, “seconds”, “minutes”, “hours”, “days”, “weeks”, “months”, and “years”. For example, if the start time is 30 minutes and the end time is 10 minutes, the query returns matching data points that occurred between the last 30 minutes up to and including the last 10 minutes. If not specified, the end time is assumed to the current date and time.
timeZoneThe time zone for the time range of the query. If not specified, UTC is used.
aggregatorsThis is an ordered array of aggregators. They are processed in the order specified. The output of an aggregator is passed to the input of the next until all have been processed. If no aggregator is specified, then all data points are returned.
tagsSystem set data point tags. Can be ori or req. See above for possible values. Tags narrow down the search. Only property values that include the tag and matches one of the values are returned. Tags is optional.
groupByThe resulting data points can be grouped by one or more tags, a time range, or by value, or by a combination of the three. The groupBy attribute in the query is an array of one or more groupers. Each grouper has a name and then additional properties specific to that grouper.
excludeTagsBy default, the result of the query includes tags and tag values associated with the data points. If excludeTags is set to true, the tags will be excluded from the response.
limitLimits the number of data points returned from the data store. The limit is applied before any aggregator is executed.
orderOrders the returned data points. Values for order are asc for ascending or desc for descending. Defaults to ascending. This sorting is done before any aggregators are executed.

You must specify either startAbsolute or startRelative but not both. Similarly, you may specify either endAbsolute or endRelative but not both. If either end time is not specified the current date and time is assumed.

Understanding Query Elements

Tags

It is possible to filter the data returned by specifying a tag. The data returned will only contain data points associated with the specified tag.

As of version 3, only the system designated tags are stored in the time-series database and can be used for filtering. User defined tags cannot be used.

The following table explains the system designated tags and their purpose:

Tag nameValuesDescription
protocolrest, mqtt, internalThe channel used to write this data point.
sourceobject, or external_clientData source type. object means the data source is the object which is writing the data, either device or app. external_client means data source is an api client which is writing data into a device or app. When sending commands to devices, source must be always an external_client.

Grouping

The results of the query can be grouped together.There are three ways to group the data; by tags, by a time range, and by value.

Grouping by Time

The time grouper groups results by time ranges. Possible unit values are “milliseconds”, “seconds”, “minutes”, “hours”, “days”, “weeks”, “months”, and “years”.

For example, you could group data by day of week. Note that the grouper calculates ranges based on the start time of the query. So if you wanted to group by day of week and wanted the first group to be Sunday, then you need to set the query’s start time to be on Sunday.

"groupBy": [
      {
        "name": "time",
        "groupCount": "7",
        "rangeSize": {
          "value": "1",
          "unit": "days"
        }
      }
]

Grouping by Value

The value grouper groups by data point values. Values are placed into groups based on a range size. For example, if the range size is 100, then values between 0-99 are placed in the first group, values between 100-199 into the second group, and so forth.

This example groups value by a range size of 100.

"groupBy": [
      {
        "name": "value",
        "rangeSize": 100
      }
]

Grouping by Bin

The bin grouper groups data point values into bins or buckets. Values are placed into groups based on a list of bin values. For example, if the list of bins is 10, 20, 30, then values less than 10 are placed in the first group, values between 10-19 into the second group, and so forth.

This example groups values into groups of 2.

"groupBy": [
      {
        "name": "bin",
        "bins": ["2", "4", "6", "8"]
      }
]

🚧

Rule

value and bin grouping can be only applied to numeric data points.

Aggregators

Optionally you can specify aggregators. Aggregators perform an operation on data points and down samples. For example, you could sum all data points that exist in 5 minute periods.

Aggregators can be combined together. For example, you could sum all data points in 5 minute periods then average them for a week period.

Aggregators are processed in the order they are specified in the query object. The output of one is send to the input of the next.

Most aggregators support downsampling. Downsampling allows you to reduce the sampling rate of the data points and aggregate these values over a longer period of time. For example, you could average all daily values over the last week. Rather than getting 7 values you would get one value which is the average for the week. Sampling is specified with a “value” and a “unit”.

Supported aggregators

Connio time-series database supports various number of aggregators.

AggregatorDescriptionParameters
avgComputes average value.See Range aggregator type
devComputes standard deviation.See Range aggregator type
countCounts the number of data points.See Range aggregator type
firstReturns the first data point for the interval.See Range aggregator type
gapsMarks gaps in data according to sampling rate with a null data point.See Range aggregator type
lastReturns the last data point for the interval.See Range aggregator type
least_squaresReturns two points for the range which represent the best fit line through the set of points.See Range aggregator type
maxReturns the largest value in the interval.See Range aggregator type
minReturns the smallest value in the interval.See Range aggregator type
percentileFinds the percentile of the data range. Calculates a probability distribution and returns the specified percentile for the distribution. The “percentile” value is defined as 0 < percentile <= 1 where .5 is 50% and 1 is 100%.percentile (double) - Percentile to count.
sumSums all values.See Range aggregator type
diffComputes the difference between successive data points.None
divReturns each data point divided by a divisor. Requires a “divisor” property which is the value that all data points will be divided by.divisor (double) - Value to divide data points by.
rateReturns the rate of change between a pair of data points. Requires a “unit” property which is the sampling duration (ie rate in seconds, milliseconds, minutes, etc...).sampling (see Sampling type) - Sets the sampling for calculating the rate. unit (see Unit type) - Shortcut for setting the sampling to a single unit. If you set the unit to SECONDS then the sampling is over one second. time_zone (Long format time zone) - Time zone for doing time calculations.
samplerComputes the sampling rate of change for the data points. Requires a “unit” which is the sampling duration (i.e. rate in seconds, milliseconds, minutes, etc...).unit(see Unit type) - Sets the sampling unit. If you set the unit to SECONDS then the sampling rate is over one second. time_zone (Long format time zone) - Time zone for doing time calculations.
scaleScales each data point by a factor. Requires a “factor” property which is the scaling value.factor (double) - Scale factor.
trimTrims off the first, last or both data points for the interval. Useful in conjunction with the save_as aggregator to remove partial intervals.trim (FIRST, LAST, BOTH) - Trims either first, last or both end data points.

Aggregator Parameters

Range aggregator type

AttributeDescription
nameAggregator name. See below.
samplingIt consists of value and unit fields. value is an integer value, and unit represents time range as “milliseconds”, “seconds”, “minutes”, “hours”, “days”, “weeks”, “months”, and “years”.
alignSamplingAn optional attribute. Setting this to true will cause the aggregation range to be aligned based on the sampling size. For example if your sample size is either milliseconds, seconds, minutes or hours then the start of the range will always be at the top of the hour. The effect of setting this to true is that your data will take the same shape when graphed as you refresh the data. This is false by default. Note that alignSampling and alignStartTime are mutually exclusive. If both are set, unexpected results will occur.
alignStartTimeAn optional attribute. When set to true the time for the aggregated data point for each range will fall on the start of the range instead of being the value for the first data point within that range. This is false by default. Note that alignSampling and alignStartTime are mutually exclusive. If both are set, unexpected results will occur.
startTimeAn optional attribute. Used along with alignStartTime. This is the alignment start time. This defaults to 0.
timeZoneTime zone to use when doing time based calculations.
"aggregators": [
{
  "name": "sum",
  "alignSampling": true,
  "alignStartTime": true,
  "sampling": {
    "value": 1,
    "unit": "minutes"
  }
}]

Unit type

Unit is a time unit represented as a string and must be one of ( MILLISECONDS, SECONDS, MINUTES, HOURS, DAYS, WEEKS, MONTHS, YEARS)

Sampling type

A sampling is a json object containing two values a value and a unit. The value is a long and the unit is see unit.

📘

Sample Size

Sample size is not the same as the result set size. It represents the number of items processed in order to generate the query result. It can be equal or greater than the result set size.

👍

Time-Series Databases

Connio's historical data API is modelled after the open source project KairosDB. Some section of this document is directly adapted from this project. We highly suggest you to take a look at the KairosDB documentation to better understand the principles behind time-series databases.

Language
Authorization
Basic
base64
:
Click Try It! to start a request and see the response here!