User Tools

Site Tools


Beta testing

What is Beta testing?

Beta testing is when beta versions of our software products are pre-released to a limited audience of users (outside of our engineering team). The goal of beta testing is to uncover any bugs and other issues by testing our software in as close to real-world scenarios as possible. Beta testing naturally follows on from alpha testing, where our software is internally tested prior to beta release.

Beta testing is an interactive process where we work with beta testers to improve our products in an iterative manner. Even the product documentation is part of the beta process, with drafts being regularly released for review along with each beta update to address any previously identified bugs or issues.

How do I get the latest beta release?

If you are an existing beta tester, running svn update on the command-line from an ESCP or N-Mesh release directory will pick up the latest beta release. See subversion for more information.

What platforms can I test on?

N-Mesh and ESCP are fully compatible with Linux, Mac OSX and Windows host platforms; and also PNaCl, Android and iOS target platforms. Makefiles for all example projects are provided. Visual Studio, Android Studio and X Code IDE project files are also provided for those who prefer building via an IDE.

How to report bugs during Beta testing?

A great way to report bugs is to use the Bugzilla system, which is an open and scalable incident tracking system used by many of the largest and most complex projects in the industry, such as the GNU-GCC and LLVM/Clang projects. Bugzilla seems to be particularly well-suited for ESCP and N-Mesh as these are also both compiler toolchains.

We recommend using Bugzilla because it helps to prevent things from “falling through the cracks”. But whether you use Bugzilla or not to report bugs during beta testing, we do require that testers provide a detailed description along with a minimal test case so we can replicate and diagnose any reported problems.

Bug writing requirements

The following information is required on a bug report to enable the problem to be quickly replicated and diagnosed:

  1. A concise description of the problem.
  2. A minimal test-case including reduced source code and makefiles1).
  3. Observed outcomes, including log files or error messages.
  4. Any other info that is relevant to the bug report that you think might be useful.

Can I use a Beta version for production work?

Alpha and Beta releases should be used for testing purposes only. Please do not use Alpha or Beta software for production builds.

When does Beta testing end?

With ESCP and N-Mesh our policy is to go live with software only when zero known bugs remain in the beta testing phase and when all principles are satisfied with the functionality and performance.

See also

The GCC project has a great explanation of minimal test cases on their site, where it states, “Most test cases can be reduced to fewer than 30 lines”.
beta.txt · Last modified: 2017/04/09 09:35 by aaron

[ Home ] [ Subversion ] [ Bugzilla ]