But with many of these industrial standard verification processes there is the classic “who watches the watchers” and “code is law” issues.
Experienced engineers, either hardware or software, may have been round this a few times themselves, but let me write it up for those who aren’t engineers, or haven’t been through this process yet.
The Issue
Many standards don’t start life as a standard. They start as an internal implementation. This isn’t bad in many ways. It’s already been bashed into something that works. The big but is that “code is law” and that original implementation is rarely opened. When doing a fresh implementation of the standard, you must match this original implementation’s behaviour. When there is a gap or ambiguousness in the standards documentation, it doesn’t matter. The implementer must match the original implementation’s behaviour. How, is the implementer’s problem/expense.
The company that created this standard rarely lets it entirely go. They own the real reference implementation. So they end up as the gatekeepers of the standard as only they can do real verification. With no competition or oversight, verification becomes expensive and low quality.
The testing is a closed process and that’s where the “who watches the watchers” issue comes in. The testing may well be wrong, but it can not be independently verified. That needs to change. If a device fails part of this test, but passes the device’s creator’s own version of that test, surely that is when more eyes are needed to make sure the gold standard test is correct? That’s not what happens though. It can’t be independently verified, the official testers, even if they had the time and inclination, may not even have access or the skills to be able to review it. The test can not be questioned. The device creator must change their device to match the official verification test.
The Solution
The first thing that should be addressed is the reference implementation. The only perfect documentation that covers all behaviour and interactions with no ambiguousness is the reference source code and schematics. They are the master document for the standard as the documentation is changed to match it, not the other way around. This is the classic “code is law“. Open reference implementations will aid developing conforming devices at lower costs.
The second thing that should be addressed is the verification testing. It’s very important they are good, and ideally done as much as possible, so can be done inexpensively during development. In the modern world “Continuous Testing” is at least the aspiration. How do we make something as good as we can and low cost? We open it. Get every developer who is going over the fine details of their implementation looking at the corresponding fine details of the tests. At that point, they are in the perfect position to watch the watchers. Even point out missing tests.
The final verification test then becomes a rubber stamp issue of running the tests one last time. If the standard body works properly, it can set up a market place of competing official test houses to drive down costs. This will aid more testing at a higher quality at a lower price.
