4.3 Specification, Documentation & Testing

Specification

Specification as we have discussed earlier describes in detail the what and why aspects of the project and its components. In other words, what is the project supposed to do functionally as part of its design and architecture as stemming from the requirements.

From a security perspective, it specifies what the assets are, where they are held, who are the actors in this context, privileges of the actors (who is allowed to access what and when), trust relationships, threat model, potential attack vectors, scenarios and mitigations.

Analyzing the specification of a project provides auditors with the above details, lets them evaluate any assumptions made and identify any shortcomings. Few smart contract projects have detailed specifications at their audit stage. At best they have some documentation about what is implemented and auditors end up spending a lot of time inferring specification from the documentation or implementation itself, which leaves them with less time for deeper vulnerability assessment.

Documentation

Documentation is a description of what has been implemented based on the design and architectural requirements. It should describe in detail how something has been designed, architected and implemented without necessarily addressing the why aspect (he design requirement goals).

Documentation in smart contract projects is typically in the form of "README" files in the GitHub repository describing individual contract functionality combined with the functional NatSpec and individual code comments.

As discussed earlier, documentation in many cases serves as a substitute for missing specification and provides critical insights into the assumptions, requirements and goals of the project team. Understanding the documentation before looking at the code helps auditors save a lot of time in inferring the architecture of the project, contract interactions, program constraints, asset flow, actors, threat model and risk mitigation measures mismatches between the documentation abd the code.

The code could indicate either stale or poor documentation and software defects or security vulnerabilities. Therefore, given this critical role of documentation, the project team is highly encouraged to document thoroughly so that auditors do not need to waste their time inferring all of the aspects by reading code instead.

Testing

Software testing or validation is a well known fundamental software engineering technique to determine if software produces expected outputs when executed with different chosen inputs. Smart contract testing has a similar motivation but is arguably more complicated despite their smaller sizes in code.

Smart contract development platforms are relatively new, with different levels of support for testing. Projects in general have very little testing before arriving to the audit stage.

Test coverage and cases give a good indication of project maturity and also provide valuable insights to auditors regarding assumptions and edge cases for vulnerability assessments. Threfore auditors should expect a very high level of testing and test coverage because it is a must-have software engineering discipline, especially with smart cotnracts that are by design exposed to everyone on the blockchain and end up holding assets worth tens or hundreds of millions of dollars.

This famous quote from Dijkstra captures the role of software testing: "program testing can be used to show the presence of bugs, but never to show their absence". This is similar to what concerns security audits.