This is a discussion of the concepts behind MTConnect's use of the HTTP protocol and the REST philosophy it has adopted. It will also discuss some best practices for implementing a client application to take advantage of Fault Tolerant capabilities built into the protocol as well as the ability to never miss a single event. We will also discuss how to use filtering to get just what you want and how best to create an application that uses multiple data streams from the same source depending on the data requirements.
MTConnect and REST
The MTConnect standard does not define its own protocol, but leverages the industry standard HTTP protocol and adds some RESTful interfaces on top of it. REST stands for REpresentational State Transfer. The term was first coined by Fielding in Chapter 5 of his thesis. The ideas of REST revolve who is responsible for state management and connections and the data for that connection, for example the last message consumed and what information the client is interested in viewing. At the time of the paper, there was a lot of debate about the best way to architect web applications and manage client session state because it severely limited server scalability, replication and recovery.
REST and the accompanying design patterns require that each request contain all the information necessary for the server to respond. The server does not need to keep any information about the clients who are making requests and just responds with the data that fulfills the client's request. This architecture has now become the defacto standard for most large web application interfaces like AWS, [Twitter] and most other web services you can think of. There are many architectural benefits to providing a REST API; for more information, read the wikipedia article on Representational state transfer.
In the case of MTConnect, the Agent has no knowledge of the clients making the request – all context must be passed in the URL and the response will contain all information the client needs to manage the results. In this way we use the REST methodology to reduce the complexity of the Agent but still provide a fully Fault Tolerant and scalable protocol.
Typical Client / Agent Conversation
The sequence of requests for a standard MTConnect conversation will typically begin with the application issuing a probe to determine the capabilities of the device. The result of the probe will provide the component structure of the device and all the available data items for each component.
Once the application determines the necessary data items are available from the Agent, it can issue a current request to acquire the latest values of all the data items and the next sequence number for subsequent sample requests. The application SHOULD also record the instanceId to know when to reset the sequence number in the eventuality of Agent failure. (See Fault Tolerance for a complete discussion of the use of instanceId). Once the current state has been retrieved, the Agent can be sampled at a rate determined by the needs of the application. After each request, the application SHOULD save the nextSequence number for the next request. This allows the application to receive all results without missing a single sample or event and removes the need for the application to compute the value of the from parameter for the next request.
The above diagram illustrates a standard conversation between an application and an MTConnect Agent. The sequence is very simple because the entire protocol is an HTTP request/response. The next sequence number handling is shown as a guideline for capturing the stream of Samples, Events, and Condition.
While the above diagram illustrates a standard conversation between an application and an MTConnect Agent, any one application or multiple applications may be having several unre-lated standard conversations occurring simultaneously with the MTConnect Agent, each poten-tially referencing different data or data types and using different portions of the Agent's data ta-ble.
The MTConnect® Agent MUST provide a probe response that describes this Agent’s devices and all the components and data items being collected. The response to the probe MUST always provide the most recent information available. A probe request MUST NOT supply any parameters. If any are supplied, they MUST be ignored. The response from the probe will be static as long as the machine physical composition and capabilities do not change, therefore it is acceptable to probe very infrequently. In many cases, once a week may be sufficient.
The probe request MUST support two variations:
- The first provides information on only one device. The device’s name MUST be specified in the first part of the path. This example will only retrieve components and data items for the mill-1 device.
The diagram below is an example of all the components and data items in relation to one another. The device has one Controller with a single Path, three linear and one rotary axis. The Controller’s Path is capable of providing the execution status and the current block of code. The device has a DataItem with type=”AVAILABILITY”, that indicates the device is available to communicate.
- The following path will request the data items for all components in mill-1 with regards to the example above (note that the path parameter refers to the XML Document structure from the probe request, not the XML Document structure of the sample):
- This is equivalent to providing a path-based filter for the device named mill-1:
- To request all the axes’ data items the following path expression is used:
- To specify only certain data items to be included (e.g. the positions from the axes), use this form:
- To retrieve only actual positions instead of both the actual and commanded, the following path syntax can be used:
19. http://10.0.1.23:3000/mill-1/sample?path=//Axes//DataItem[@type=”POSITION” and @subType=”ACTUAL”]
20. http://10.0.1.23:3000/mill-1/sample?path=//Axes//DataItem[@type=”POSITION” and @subType=”ACTUAL”]&from=50&count=100
The above example will retrieve all the axes’ positions from sample 50 to sample 150. The actual number of items returned will depend on the contents of the data in the Agent and the number of results that are actual position samples.
All parameters MUST only be given once and the order of the parameters is not important. The MTConnect® Agent MUST accept the following parameters for the sample request:
path - This is an XPath expression specifying the components and/or data items to include in the sample. If the path specifies a component, all data items for that component and any of its sub-components MUST be included. For example, if the application specifies the path=//Axes, then all the data items for the Axes component as well as the Linear and Rotary sub-components MUST be included as well.
from - This parameter requests Events, Condition, and Samples starting at this sequence number. The sequence number can be obtained from a prior current or sample request. The response MUST provide the nextSequence number. If the value is 0 the first available sample or event MUST be used. If the value is less than 0 (< 0) an INVALID_REQUEST error MUST be returned.
count - The maximum number of Events, Condition, and Samples to consider, see detailed explanation below. Events, Condition, and Samples will be considered between from and from + count, where the latter is the lesser of from + count and the last sequence number stored in the agent. The Agent MUST NOT send back more than this number of Events, Condition, and Samples (in aggregate), but fewer Events, Condition, and Samples MAY be returned. If the value is less than 1 (< 1) an INVALID_REQUEST error MUST be returned.
interval – The Agent MUST stream Samples, Events, and Condition to the client application pausing for interval milliseconds between each part. Each part will contain a maximum of count Events, Samples, and Condition and from will be used to indicate the beginning of the stream.
The nextSequence number in the header MUST be set to the sequence number following the largest sequence number (highest sequence number + 1) of all the Events, Condition, and Samples considered when collecting the results.
If no parameters are given, the following defaults MUST be used:
The count MUST default to 100 if it is not specified.
If specified without the at parameter, the current request retrieves the values for the components’ data items at the point the request is received and MUST contain the most current values for all data items specified in the request path. If the path is not given, it MUST respond with all data items for the device(s), in the same way as the sample request. The current MUST return the values for the data items, even if the data items are no longer in the buffer.
Current MUST return the nextSequence number for the event or sample directly following the point at which the snapshot was taken. This MUST be determined by finding the sequence number of the last event or sample in the Agent and adding one (+1) to that value. The nextSequence number MAY be used for subsequent samples.
The Samples, Events, and Condition returned from the current request MUST have the time-stamp and the sequence number that was assigned at the time the data was collected. The Agent MUST NOT alter the original time, sequence, or values that were assigned when the data was collected.
http://10.0.1.23:3000/mill-1/current?path=//Axes//DataItem[@type=”POSITION” and @subType=”ACTUAL”]
This example will retrieve the current actual positions for all the axes, as with a sample, except with current, there will always be a sample or event for each data item if at least one piece of data was retrieved from the device.
http://10.0.1.23:3000/mill-1/current?path=//Axes//DataItem[@type=”POSITION” and @subType=”ACTUAL”]&at=1232
The above example retrieves the axis actual position at a specific earlier point in time - in this case, at Sequence Number 1232.
path - same requirements as sample.
interval - same requirements as sample. MUST NOT be used with at.
at - an optional argument specifying the MTConnect protocol sequence number. If supplied, the most current values on or before the sequence number MUST be provided. If at is not provided, the latest values MUST be provided. at MUST NOT be used with the interval as this will just return the same data set repeatedly.
Getting the State at a Sequence Number
The current at allows an application to monitor real-time conditions and then perform causal analysis by requesting the current values for all the data items at the sequence number of interest. This removes the requirement that the application continually poll for all states and burden the server and the network with unneeded information associated with faults or other abnormal conditions.
The following table is a list of events and conditons:
When the interval parameter is provided, the MTConnect® Agent MUST find all available events, samples, and condition that match the current filter criteria specified by the path delaying interval milliseconds between data or at its maximum possible rate. The interval indicates the delay between the end of one data transmission and the beginning of the next data transmission. A interval of zero indicates the Agent deliver data at its highest possible rate.
The interval MUST be given in milliseconds. If there are no available events or samples, the Agent MAY delay sending an update for AT MOST ten (10) seconds. The Agent MUST send updates at least once every ten (10) seconds to ensure the receiver that the Agent is functioning correctly. The content of the streams MUST be empty if no data is available for a given interval.
The MTConnect agent is capable of storing a limited number of assets. An Asset is something that is associated with the manufacturing process that is not a component of a device, can be removed without detriment to the function of the device, and can be associated with other devices during their lifecycle.
The assets are referenced by their asset id. The id is a permanent identifier that will be associated with this asset for its entire life.
When an asset is added or modified, the agent will generate an AssetAdded event or an AssetModified event for the device. For devices that manage assets, these data items MUST be added to the data items list at the device level.
The agent MUST store one or more assets. It MAY decide the capacity based on the available storage and scalability of the implementation. Similar to Events, Samples, and Conditions, the data will be managed on a first in first out basis.
The asset’s timestamp will be used to determine the oldest asset for removal. As assets are modified, their timestamp is updated to the current time. As with all timestamps in MTConnect, the time will be given using the UTC (or GMT) timezone.
HTTP Response Codes and Error
MTConnect® uses the HTTP response codes to indicate errors where no XML document is returned because the request was malformed and could not be handled by the Agent. These errors are serious and indicate the client application is sending malformed requests or the Agent has an unrecoverable error. The error code MAY also be used for HTTP authentication with the 401 request for authorization. The HTTP protocol has a large number of codes defined ; only the following mapping MUST be supported by the MTConnect® Agent:
|200||OK||The request was handled successfully|
|400||Bad Request||The request could not be interpreted|
|500||Internal Error||There was an internal error in processing the request. This will require technical support to resolve|
|501||Not Implemented||The request cannot be handled on the server because the specified functionality is not implemented|
||The request did not have sufficient permissions to perform the request|
||The device specified in the URI could not be found|
||The sequence number was beyond the end of the buffer|
||The count given is too large|
||The URI provided was incorrect|
||The request was not one of the three specified requests|
||Contact the software provider, the Agent did not behave correctly|
||The XPath could not be parsed. Invalid syntax or XPath did not match any valid elements in the document|
||A valid request was provided, but the Agent does not support the feature or request type|
||An asset ID cannot be located|
For a full list of HTTP response codes see the following document: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
When an MTConnect® Agent collects information from the device, it assigns each piece of information a unique sequence number. The sequence number MUST be assigned in monotonically increasing numbers in the order they arrive at the Agent. Each source SHOULD provide a time-stamp indicating when the information was collected from the component. If no time-stamp is provided, the Agent MUST provide a time-stamp of its own. The time-stamps reported by the Agent MUST be used as the means for the ordering of the messages as opposed to using the sequence number for this purpose.
Note: It is assumed the time-stamp is the best available estimate of when the data was recorded.
If two data items are sampled at the same exact time, they MUST be given the same time stamp. It is assumed that all events or samples with the same timestamp occurred at the same moment. A sample is considered to be valid until the time of the next sample for the same data item. If no new samples are present for a data item, the last value is maintained for the entire period between the samples. Important: MTConnect® only records data when it changes. If the value remains the same, MTConnect MUST NOT record a duplicate value with a new sequence number and time stamp. There MUST NEVER be two identical adjacent values for the same data item in the same component.
For example, if the Xact is 0 at 12:00.0000 and Yact is 1 at 12:00.0000, these two samples were collected at the same moment. If Yact is 2 at 12:01.0000 and there is no value at this point for Xact, it is assumed that Xact is still 0 and has not moved.
The sequence number MUST be unique for this instance of the MTConnect® Agent, regardless of the device or component the data came from. The MTConnect® Agent provides the sequence numbers in series for all devices using the same counter. This allows for multi-device responses without sequence number collisions and unnecessary protocol complexity.
As an implementation warning, it is the applications responsibility to make sure it does not miss information from the Agent. The Agent has no awareness of the application or its requirements for data, and it therefore does not guarantee the application receive all pieces of data. The Agent protocol makes it easy for the application developers to determine if they have received all pieces of data by scrolling through the buffer. If they ever receive an OUT_OF_RANGE error due to providing a from argument that references a sequence number prior to the beginning of the retained data, they know they have missed some information.
If the application only uses current requests, it may miss information since it will only be receiving a snapshot at various points in time. For some display application that do not need to store or reason on the data, this may be adequate, but if more in-depth analysis is to be performed, it is advised that the application make requests based on their data requirements using filtering and streams to get all vital information. For example, the application can request all condition types and controller events, and then sample other pieces of data for which they have less strict requirements. Breaking things out like this will allow for continuous data flow and minimal bandwidth utilization.
The application may request any sequence of data within the buffer at any time using either the sample from or the current at semantics. With these two calls it is easy for the application to go back in time and find data prior to an occurrence. It is of course limited to the size of the buffer and rate of incoming data.