Author: James Butcher, IOTech Product Manager | EdgeX Foundry QA/Test Chair
Edge computing is all about the premise of an organization processing its operational data at the edge of the network. That means performing data collection functions and running decision making and control algorithms as close to the edge sensors and OT devices as possible.
The EdgeX Foundry framework helps makes that premise a reality and provides an open, vendor-neutral approach for doing so. However, we must never lose sight of the fact that businesses will want to understand the full implications, including the fully loaded cost, of rolling out an edge system.
The EdgeX QA & Test Working Group
The EdgeX QA & Test working group is responsible for ensuring that EdgeX releases meet the high standards of reliability and robustness that are required at the Industrial IoT edge. Our remit includes verifying the framework is well tested and ensuring we have the processes in place to maintain and monitor these high standards.
Solid Testing as Standard
EdgeX Foundry is an open-source community project, but we take testing and reliability very seriously. We have developed our own EdgeX test framework called TAF. You’ll hear us speak about TAF a lot – it stands for Test Automation Framework and supports the functional, integration and performance testing needs that we have across the project. A version of EdgeX is only released when the TAF reports all tests are passing, for example.
EdgeX is transitioning to a second version of its APIs so there has been a big emphasis on ensuring the V2 API is accurately tested. TAF has been invaluable here because we have been able to easily update and add new tests for the new API. We are pushing ahead for the first release of the new APIs in the EdgeX Ireland release due in the spring.
Gathering Performance Metrics
The QA & Test group also record and gather performance metrics associated with the usage of the EdgeX platform. A common question from EdgeX users is what sort of hardware is EdgeX aimed at and how fast can it perform? Well, of course, EdgeX has been designed to be hardware and operating system independent, but key metrics like footprint, CPU usage and latency of data flow do need to be provided so users can plan which IoT edge gateways and edge servers to use. Sometimes users have the luxury of being able to select brand new hardware to deploy in their edge projects, but often it is necessary to re-use existing already-deployed hardware.
Over the last release cycle, culminating in the EdgeX Hanoi release, the QA & Test group have put renewed focus on gathering and reporting these performance metrics. We have also improved the accuracy of the figures by increasing the number of tests ran and stating metric averages along with minimum and maximum readings, etc. Note also that our TAF system can monitor performance regressions. We’ve added threshold testing such that fails are reported if a code commit adversely affects any of the performance metrics. These are all tools to ensure that EdgeX quality and performance remains as high as possible as the project evolves.
Presenting the Performance
We’ve decided to go beyond just providing the raw metrics and instead have created a report that presents the key numbers along with some comment and explanation. Please download the Hanoi Performance Document.
I hope you find this information useful. We are aiming to provide an updated version of this document for each EdgeX release and will track the progression as we go.
Commercial Support is Key
EdgeX is a fantastic project for helping to simplify the challenges of edge computing. The open-source and collaborative nature of the project brings together ideas from edge experts around the world. However, to pick up the point about businesses needing to understand the full implications of selecting an edge solution, companies are usually reluctant to deploy open-source code without having guaranteed help at hand and support contracts in place.
So, switching to my other role in the EdgeX ecosystem for a moment, let me mention how companies like IOTech provide products and services to mitigate that risk…
I am the product manager of Edge Xpert which is IOTech’s value-add and commercially supported implementation of EdgeX. Our Edge Xpert 1.8 release is based on Hanoi and adds another level of features and services to help you. Head over to our webpage at https://www.iotechsys.com/products/edge-xpert/ to read about how Edge Xpert can get your EdgeX systems to market much quicker and with less risk.
Future Plans
Obviously quality and robustness is very important to me in both of my EdgeX roles. Testing is never finished of course, and we are always looking to add to our test and performance tracking capability. Future iterations of the performance testing will include scalability metrics and performance associated with the EdgeX Device Services – for example, how fast can Modbus data be collected, and how many data points can be received in a certain period, etc. These factors are all important in understanding how the platform will scale as it is deployed and understanding the number of deployments - and thus gateways and edge servers that will be needed.
Open Discussions
Feel free the join the weekly EdgeX QA & Test calls where we discuss progress and issues that need to be addressed. We meet every Tuesday at 8am PST. You can find the meeting links on our page here
Read more about the Hanoi Release