========================================= Setup JUnit 5 Tests in RapidWright ========================================= RapidWright uses JUnit 5 for Unit Testing. This article aims to give an overview about how to run tests, as well as how to write your own. Running the Tests ------------------ All testcases are located in the ``test/`` directory. JUnit does not need a central list of testcases. Instead, it searches the directory for all classes that contain tests. Tests are marked by annotations (see later). After builing a list of all tests, it executes them one by one. Some tests depend on DCPs which are stored in a Git submodule --- a feature that allows a specific commit of another Git repository to exist as a subdirectory of the current repository. To check out the specific commit of a submodule, run:: git submodule update --init from the parent RapidWright repository where ``--init`` is only strictly necessary (but harmless otherwise) on the first invocation. To run the tests via Gradle, use the task ``test`` or ``build`` (which depends on ``test``). After running the tests, Gradle will output the results both as an HTML document in ``build/reports/tests/test*`` and as JUnit-internal XML in ``build/test-results/test*``. Note that Gradle knows whether the input to the tests changed and will not rerun them if they are up to date. There is integration for JUnit in all major IDEs. When loading RapidWright into your IDE, you should set ``test`` as source directory for tests. Your IDE should allow you to run all the tests or choose a single class to run. Alternatively, one can execute Gradle from the command line with the ``testJava`` or ``testPython`` task to run all tests, and restricted to specific tests with one or more ``--tests `` arguments. For example:: ./gradlew testJava --tests com.xilinx.rapidwright.design.* --tests *PartNameTools.testGetPartCase would run all Java test methods under all classes within the ``com.xilinx.rapidwright.design`` package, as well as the single test method ``testGetPartCase`` from the ``com.xilinx.rapidwright.device.TestPartNameTools`` class. Note that the ``test`` task depends on ``testJava`` and ``testPython`` but does not support filtering. Writing Testcases ------------------ JUnit uses Annotations to tag methods as testcases. While there are more specialized annotations, most testcases will be tagged with the annotation ``@Test`` (from the ``org.junit.jupiter.api`` package). A test class with a single (empty) test method might look like this: .. code-block:: java import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; public class MyTestClass { @Test public void test() { } } Test methods should be ``public`` and cannot be ``static`` and not have parameters. JUnit will create an instance of the class, so the class cannot have any constructor parameters. Testcases communicate failures by throwing an exception. JUnit will then mark it accordingly. Instead of using an ``if`` to check for something and then manually creating an exception, you can use the ``Assertions`` class (from the package ``org.junit.jupiter.api``). It offers convenience methods for often used checks: - ``assertEquals`` - ``assertArrayEquals`` - ``assertNotEquals`` - ``assertSame`` All these methods have a parameter for an expected value and an actual value. Optionally, a message parameter can be passed to explain what part of the test encountered an error. A very simple test to check that addition works as expected might look like this: .. code-block:: java import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; public class MyTestClass { @Test public void test() { Assertions.assertEquals(2, 1 + 1); } } Parameterized Tests ------------------- Normal test methods do not have parameters. If you want to run the same test on a range of data, you can use a loop. However, once the test fails for one set of data, the whole testcase execution is over. Data after the first failure will not be run. JUnit allows parameters on testcases. They are marked with ``@ParameterizedTest`` instead of ``@Test``. The annotation has an optional parameter (``name``) that allows you to override the generated test's name to make it more descriptive. You need to specify a source for values for these parameters. One option is use a separate method that return a ``Collection`` or ``Stream``. One instance of ``Arguments`` describes one invocation of the testcase method. The value source is specified as another annotation (here: ``@MethodSource``). A simple example that calls ``testNonzero(int i)`` on all numbers from 1 to 10: .. code-block:: java import java.util.stream.IntStream; import java.util.stream.Stream; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.Arguments; import org.junit.jupiter.params.provider.MethodSource; public class MyTestClass { @ParameterizedTest(name = "Check that {0} is nonzero") @MethodSource() public void testNonzero(int i) { Assertions.assertNotEquals(0, i); } public static Stream testNonzero() { return IntStream.rangeClosed(1, 10).mapToObj(i -> Arguments.of(i)); } } RapidWright-specific Considerations ----------------------------------- RapidWright's tests are automatically run on Github Actions. There are rather strict restrictions in terms of maximum memory (7GB) and some parts of RapidWright can exceed that limit. You should keep this limitation in mind while writing testcases: - Testcases should be limited to a single ``Device``. If you have to use multiple Devices, take care that only one Device is referenced at the same time. - When instantiating a ``Design``, use a small ``Device`` for it. To identify issues with files being left open, there is a JUnit extension that compares the list of open files before and after a testcase. It will fail the testcase if there are changes. This extension (``com.xilinx.rapidwright.support.CheckOpenFilesExtension``) is automatically registered when JUnit tests are run with Gradle. Testcase DCPs ^^^^^^^^^^^^^ Tests requiring new DCP(s) will need to fork the `RapidWrightDCP `_ repository to gain write permissions. The DCP(s) to be added should have no encrypted components inside and the EDIF inside the DCP should be readable (not encrypted). A readable EDIF file can be generated using Vivado either `automatically `_ upon load in RapidWright or via ``write_edif`` (see :ref:`RapidWright and Design Checkpoint Files`). Use the ``ReplaceEDIFInDCP`` tool to replace the EDIF inside a DCP, for example: .. code-block:: bash rapidwright ReplaceEDIFInDCP design.dcp readable_design.edf will replace the EDIF file inside ``design.dcp`` with ``readable_edif.edf``. Next, execute the following:: # Add, commit, push new DCP(s) into new branch on fork cd test/RapidWrightDCP git remote add fork https://github.com//RapidWrightDCP # Only necessary first invocation git checkout -b git add git commit git push -u fork cd ../.. # Commit new submodule reference git commit test/RapidWrightDCP -s -m "(Description)" The submodule can now be used as a regular Git repository during development; remember to commit new submodule references from the RapidWright repository using:: git commit test/RapidWrightDCP -s -m "(Description)" Once ready, please create new pull requests in both the upstream RapidWright and RapidWrightDCP repositories. When both pull requests have been approved, the following situation will be present:: RapidWrightDCP (upstream) ... o--o---------------x \ / (PR#123) RapidWrightDCP (fork) ... o--o ... o--o ^ (commit `abc`) RapidWright (upstream) ... o--o---------------x \ / (PR#456) RapidWright (fork) ... o--o ... o--o ^ (submodule refers to commit `abc` on RapidWrightDCP fork) Here, RapidWright's ``PR#456`` refers to commit ``abc`` which is present only on the fork. The expectation would be that the RapidWrightDCP's ``PR#123`` would be merged first after which ``PR#456`` can then update its RapidWrightDCP submodule reference to include upstream's newly merged result:: RapidWrightDCP (upstream) ... o--o---------------o (commit `def` including \ / PR#123) RapidWrightDCP (fork) ... o--o ... o--o RapidWright (upstream) ... o--o------------------o \ / / (PR#456) RapidWright (fork) ... o--o ... o--o--o ^ (submodule updated to commit `def` on RapidWrightDCP upstream) This submodule reference can be updated back to upstream as follows:: # Return submodule to upstream master cd test/RapidWrightDCP git checkout master git pull cd ../.. # Commit new submodule reference git commit test/RapidWrightDCP