Best practices in blockchain testing
The need to create defect-minimized software has never been more keenly felt than in blockchain development. Unlike web platforms, where a ‘move fast and break things’ mentality might be acceptable, such an approach is not an option to entities where data is intrinsically linked to monetary value.
At Moonsong Labs, we commit significant mindshare and resources to rigorous testing, a practice that not only solidifies our reputation with an industry-leading approach but also contributes positively to the broader landscape of blockchain development. Given blockchain doesn’t have “outages” as traditional software may, having robust and ongoing blockchain testing in place allows sustainability in development without sacrificing security.
This article will delve into our testing methodologies, including production state monitoring, fuzzing, and audits, illustrating how we’ve tailored these practices to blockchain’s unique demands, while highlighting examples of blockchain testing within Polkadot’s ecosystem. While we work with many technology stacks like Rust, EVM, and Zero Knowledge, highlighting Polkadot is a choice driven by its innovative design in facilitating multi-chain interoperability and its growing significance in the Web3 landscape, which presents unique testing challenges and opportunities.
The basics of blockchain testing
Before diving into the blockchain testing practices Moonsong Labs has developed, having a solid understanding of basic blockchain terms coupled with a thorough testing process during development, is vital to delivering quality software.
Below are some common terms and components important to keep in mind with developing and testing blockchains. These components make blockchain systems function and are key considerations when designing secure and reliable software.
Definitions
- State: This is the current values of the data, i.e. Account balances, smart contracts deployed, etc. When you interact with the blockchain you are interacting with its State. State tests look at what’s been added to the chain and whether it makes sense (i.e. is circulating supply > tokens minted)
- State Transition Function: This is the business logic behind the blockchain. It is the code which says you cannot transfer more coins than you own, which is implemented in the State Transition Function.
- RPC: This functionality relates to how third parties connect to the blockchain, and the kind of requests they can make. An RPC endpoint is analogous to traditional API testing in the web2 world.
- Ledger: A ledger is a list of transactions, which is the blockchain itself. Looking at the block history, you are able to read the ledger to verify the current state is correct, but to see historic snapshots of state – you require an indexer (a service which provides a database layer in between the blockchain and queries).
- P2P: Peer-to-Peer relates to the networking aspect of a blockchain network, which includes the Gossip network, which is how transactions sent to any node can propagate through the entire network.
Now that we have covered basic definitions of blockchain components, we will cover basic phases of developing and testing the software. Please note this is a high-level overview of the general approach we take at Moonsong Labs and each project has uniquely tailored solutions based on the scope of work.
Phases
-
Ideation, experimentation
- Start by understanding the requirements and objectives of the blockchain application.
- Brainstorm and ideate on how the blockchain will be used to solve specific problems or provide value.
- Experiment with different blockchain technologies and frameworks to determine which one best fits your needs.
-
Design + Setting up CI
- Design the architecture of your blockchain solution, including smart contracts, consensus mechanisms, and data structures.
- Set up a Continuous Integration (CI) pipeline for automated testing and deployment of your blockchain code.
- Ensure that the CI pipeline includes comprehensive testing suites for different aspects of your blockchain, such as smart contracts, node software, and APIs.
-
Implementation of feature
- Develop the blockchain features according to the design and requirements.
- Implement smart contracts, consensus algorithms, and other components of the blockchain.
-
Testing of feature
- Perform various types of testing, including unit testing, integration testing, and security testing, on the implemented blockchain features.
- Test for functionality, performance, and security vulnerabilities.
- Ensure that the smart contracts are thoroughly tested to prevent vulnerabilities like reentrancy or overflows.
-
Repeating 3&4 until milestone
- Iteratively implement and test features in smaller milestones or sprints.
- Each milestone should include design, implementation, and testing phases.
- Validate that the implemented features align with the project’s goals and meet the predefined criteria before proceeding to the next milestone.
-
Repeat 2
- Continuously update and improve your CI pipeline as you discover new issues or requirements during the development and testing process.
- Maintain a strong focus on automation to streamline testing and deployment.
Implementing streamlined development and testing processes empowers Moonsong Labs to deliver quality software from ideation to execution.
A note on Quality Vs Testing in blockchain development
The trend seen in modern software development is the discernible shift towards emphasizing software quality and not merely finding bugs. Whilst this may initially seem like a mere semantic play, it actually reveals a change in tact. Testing, traditionally seen as an activity to uncover defects, is the first thing Moonsong Lab’s does to ensure a product’s market readiness. However, as we have all found, one person’s feature is another one’s bug. Designing for quality from the first initial phase of R&D can significantly reduce the likelihood of expensive refactors.
The maxim to follow is that Diversity breeds Robustness akin to concept hybrid vigour in biology. To achieve comprehensive quality assurance, a multi-spectrum battery of test suites is essential. We use a multitude of testing practices, whereas many use only two types, allowing us to catch more issues up front rather than down the line. The rest of this article will focus on various testing activities, but it’s important to note that this represents only a subset of the practices we follow to optimise for the highest quality software.
Testing pyramid applied to blockchain development
There is the truism that the more realistic a test, the more fragile and challenging it becomes to maintain. A full end-to-end test, especially if designed poorly, is prone to break with each redesign of the front-end, despite the fact that users might adapt seamlessly to these changes.
It would be tempting to hope that with enough unit tests you could model every scenario. However, this is a fallacy. Unit tests, by their very nature, are confined to the most granular level and do not encompass many complexities, like integration; It is crucial to recognize that modeling the integration is just as important and the misuse of an API can still result in defects from an end user’s perspective.
A visualization of the typical balance that is struck is the testing pyramid, a concept created by Mike Cohn in his book Succeeding With Agile:
Whilst the above model serves a valuable reference, it has not been the testing approach we have taken at Moonsong Labs. Recognizing that there is no one-size-fits-all solution in software testing, this model has primarily served as a foundational guide to balance cost efficiency versus safety in testing. However, in the dynamic realm of Web3, and more specifically with PolkadotSDK the landscape is far different and presents a distinct challenge that requires an alternate strategy.
Polkadot stands out in the Web3 landscape for its pioneering approach to blockchain interoperability and scalability. Its framework allows for the creation of parachains—individual blockchains that operate in parallel within the Polkadot ecosystem and are connected to the central relay chain. This architecture not only enhances scalability but also introduces complex interdependencies and interactions that are crucial to test thoroughly.
At Moonsong Labs, working on Moonbeam within the Polkadot ecosystem has meant adapting our testing strategies to address these intricacies. The interconnected nature of parachains, their interaction with the relay chain, and the need for consistent updates for compatibility, necessitate a more nuanced and flexible approach to testing. We have been able to take these learnings and apply them to the rest of our clients.
Challenges of building in blockchain with an open source stack
Open source development comes with its own inherent pitfalls. A pertinent issue we focus on at Moonsong Labs is navigating the dependency graph associated with importing libraries into a given project – be that Rust crates or JavaScript NPM packages. This is not a case of laziness, but it is in fact a best practice. Utilizing audited and battle-tested libraries, which embody collective expertise, is often the safer and more prudent choice in the build versus buy decision.
The challenge then becomes, how do you ensure that when updating a library you do not break compatibility with not only your own code, but other packages you may depend on. Whilst it may be tempting to avoid this problem by updating very infrequently, such a strategy will only defer integration headaches since mandatory security upgrades force the decision at some point in the future.
ℹ️ With Polkadot parachains, this option is wholly unavailable. Maintaining compatibility with the relaychain requires parachain teams to update dependencies on every release; otherwise one courts disaster (failure to finalize blocks).
The answer here lies in striking a balance between striving for the most robust automated testing whilst keeping the maintenance burden controllable so that it does not impede future growth. Achieving this is easier said than done, and requires the automated test systems to be engineered with the same amount of diligence and nous as the software-under-test itself. It is important to view this task not as a one-time effort but as an ongoing commitment in the cost-to-participate. Human readable testing is also top of mind with our process for blockchain testing which aids in the following:
Improved Maintainability: Human-readable testing significantly enhances the maintainability of our testing systems. By crafting tests that are easily understood, we facilitate quicker updates and adaptations, essential in the rapidly evolving blockchain environment. This approach allows our team to efficiently manage and update tests in line with new developments or changing requirements, thereby ensuring our testing framework remains as agile and resilient as the software it tests.
Better Documentation: Our focus on human-readable testing naturally leads to improved documentation. These types of tests double as high-quality documentation, providing clear and comprehensive insights into the testing process and intentions. This clarity is invaluable not only for current team members but also for future collaborators who may need to understand the evolution and rationale of our testing strategies.
Enhanced Error Identification and Resolution: With human-readable tests, identifying and resolving errors becomes a more streamlined and efficient process. Clear, understandable tests allow our developers to quickly pinpoint the source of a problem and understand the context around it. This clarity accelerates the troubleshooting process, leading to quicker resolutions and a more robust final product.
In essence, at Moonsong Labs, our commitment to human-readable testing within our robust automated testing framework is a strategic choice.
A Note on Polkadot Parachains
When deciding which ecosystem to build your blockchain, choosing to be a Polkadot parachain comes with manifold benefits: large economic security, first class decentralization (i.e. robustness), native interoperability. However, in exchange for all of these benefits, parachain teams are obliged to participate in certain idiosyncratic ceremonies, such as on-chain upgrades.
This feature is a super power, which not only allows for forkless upgrades of your chain – but is done in such a way that enables the relaychain to judge the validity of blocks submitted.
However, botching the upgrade process can leave your own chain in a stuck state. If it is producing bad blocks – not only will your chain not work, but then you are also then unable to submit a patch to fix the chain. All is not lost, and there are paths to recovery (which fall beyond the scope of this article), but the key takeaway is the importance of minimizing risks at every juncture. A big part of this risk mitigation process is the commitment to continual testing.
At Moonsong Labs, we have developed 3 separate automated processes:
- via Chopsticks, Zombienet, in-house ForkTesting
- 6 testnets
- multiple rounds of manual testing for each new runtime
Our focus is optimising for safety and liveness, recognising the importance of robust assurance that upgrades are applied without surprises.
Production state monitoring in blockchain development
So far we have covered verifying the business logic of a blockchain, commonly referred to as the State Transition Function. Equally crucial is the ongoing monitoring and testing of the data that has been recorded, ensuring it abides by the rules we expect it to. This responsibility is significant but provides an early warning system to guard against emergent defects that may occur.
In the broader software industry, this is a practice common in the ‘Shift Right’ mindset, which involves the live monitoring of the production environment. At Moonsong Labs, we can achieve this by establishing a set of invariants that are continuously validated via live end-to-end calls to the blockchain that assert assumptions we have.
The above is an example of a basic validation between different pallet storages to ensure that writes have a consistency across them. There are also far more involved checks whereby we verify that every smart contract deployed abides by preprogrammed constraints, fees are correctly calculated and reserve balances are being properly updated – to name a few.
This monitoring effort at Moonsong Labs also applies to foreign chains we have links with via Cross-Chain Messaging (XCM), and has been incredibly useful in flagging when outages, bugs or improper calls have been received from external chains.
What about Fuzzing in blockchain development?
Fuzzing is a testing practice where randomized test inputs are given to the software being evaluated. The outputs from these tests are inspected against test invariants or system malfunctions. This method is effective in uncovering hidden issues that might not surface during conventional testing scenarios.
This is incredibly useful for blockchain development, where network liveness is especially vulnerable to malicious calls which can stall block production due to bugs that result in system exceptions. Fuzzing excels in uncovering issues that may allude even the most meticulous human, as it can execute tens or hundreds of thousand iterations. It will find many issues that no sane human would be able to find, and is typically able to perform without manual intervention. The capability makes fuzzing an indispensable tool in testing automation as it can ensure resilience and security of blockchain networks.
The probabilistic nature of this testing approach makes it an essential component in any software testing toolkit, and one which we actively employ at Moonsong Labs, particularly to subsume the activities of exploratory testing of our core functionality. It is worth bearing in mind that this is by no means a panacea – not all testing can be replaced by fuzzing alone.
While Fuzzing is a powerful tool capable of generating a vast array of results, the real challenge lies in the analysis of these outcomes. A significant amount of time can be invested in configuring the testing framework to be sufficiently set up to detect real issues while also being able to minimize the noise produced by false negatives. The process of this fine-tuning often becomes a central focal point for many development teams.
What about Audits in blockchain testing?
Code auditing is another vital part of any blockchain team preparing to deploy their code to production. Having another fresh set of eyes inspect your code is valuable, but having those eyes belong to a security specialist doubles the value.
Engaging with external auditing firms is not just a service arrangement; it’s a partnership where understanding the nuances of code auditing is key to deriving maximum benefit.
Audits, which involve a significant manual component are conducted on a specific snapshot of the code. Continual updates to the software will render previous audits moot, so it makes sense to be done at the end of the delivery cycle, not after every code commit.
Rather than a blunt tool, to be applied to every code commit – harnessing the expertise and time of auditors requires the understanding of this practice and how it can complement the security practices already employed. Code audits cannot replace thorough testing throughout the development lifecycle of a technology product.
Closing note
A common mistake that many commit in software development is: thinking that tests that break are indicative of either a flawed test, or problematic change which is inherently bad. At Moonsong Labs, we challenge this view and see broken tests as opportunities for refinement and improvement.
Broken tests often highlight a crucial area that needs to be addressed, such as:
- Pointing to a mismatch between the expected behaviour and the actual outcome, necessitating an adjustment in test expectations
- Revealing a discrepancy in the acceptance criteria, requiring a modification to the software itself
The balance must be struck as the cost of over-optimising for safety too early brings too much friction to new development. Continual efforts should be made to ensure that testing is fluid and that it is cost-effective and easy for humans to understand.
With this testing approach, quality assurance cannot be relegated to merely an afterthought – but must be an integral part of the DNA of how you deliver software.
After all, if “If it isn’t tested, then it doesn’t work.”
Thus, our approach to blockchain testing is fundamental to building rigorously tested software that is secure, trustworthy and robust. We stand by the principle that good testing equates to good products, a philosophy that continually guides our efforts in delivering exceptional blockchain solutions.