261 lines
10 KiB
Plaintext
261 lines
10 KiB
Plaintext
===========================================================================
|
|
RISC-V Torture Test Generator
|
|
===========================================================================
|
|
# Author: Yunsup Lee and Henry Cook
|
|
# Date: January 29th, 2012
|
|
# Version: (under version control)
|
|
|
|
This is the RISC-V torture test generator and framework. This repository
|
|
contains three sub-projects that build upon one another. The first,
|
|
[generator], is used to create a single random torture test. The second,
|
|
[testrun], is used to run a particular test on particular simulators,
|
|
diffing the resulting signature with the ISA simulator and optionally
|
|
creating a derivative test subset that pinpoints the divergence. The third,
|
|
[overnight], wraps testrun, allowing tests to be run repeatedly for a given
|
|
duration or until a failure count.
|
|
|
|
|
|
---------------------------------------------------------------------------
|
|
Instructions
|
|
---------------------------------------------------------------------------
|
|
|
|
Modify "config/default.config" to set the parameters desired for building tests
|
|
(e.g., setting which instructions to use and in which ratio).
|
|
|
|
Modify "Makefile" as desired to execute the C simulator or RTL simulator of
|
|
your choice, and to set the other parameters as you require.
|
|
|
|
To build a single test and test it on Spike:
|
|
|
|
$ make igentest
|
|
|
|
To build single test and run it on the C simulator or RTL simulator, use
|
|
"make cgentest" or "make rgentest".
|
|
|
|
To run overnight tests, you can use "make cnight" and "make rnight".
|
|
|
|
---------------------------------------------------------------------------
|
|
Signatures
|
|
---------------------------------------------------------------------------
|
|
|
|
Torture works by dumping the register state out to memory at the end of the
|
|
test program execution. This output is then compared against the output from
|
|
the Spike ISA simulator.
|
|
|
|
The torture program writes the register state to the memory address specified
|
|
by "xreg_output_data", which is located in the memory section
|
|
".global begin_signature". The Spike ISA simulator will write out the data
|
|
found in the "begin_signature" section on exit if provided with the
|
|
"+signature=" argument:
|
|
|
|
$ spike +signature=my_spike_signature.txt test_binary
|
|
|
|
The Rocket-chip infrastructure uses the "riscv-fesvr" program to control the
|
|
execution of the C and RTL simulators. The "riscv-fesvr" also accepts the
|
|
+signature argument too.
|
|
|
|
$ ./csim-rocket-chip +signature=my_rocket_signature.txt test_binary
|
|
|
|
A simple diff between the Spike and chip simulator signatures will tell you if
|
|
any errors have occurred.
|
|
|
|
$ diff my_spike_signature.txt my_rocket_signature.txt
|
|
|
|
|
|
**PORTING TORTURE TO YOUR OWN RISC-V PROCESSOR:**
|
|
|
|
If you would like to use riscv-torture with your own RISC-V processor, you will
|
|
need to provide a way to dump the "begin_signature" section to a file.
|
|
|
|
---------------------------------------------------------------------------
|
|
Low-level Usage
|
|
---------------------------------------------------------------------------
|
|
|
|
Some basic use cases are illustrated here (note the Makefile abstracts this for
|
|
you).
|
|
|
|
Make a single test:
|
|
% ./sbt generator/run
|
|
% cd output
|
|
% make
|
|
% spike +signature=test.sig test
|
|
|
|
Take an existing test and diff the signatures of ISA and C simulators:
|
|
% ./sbt 'testrun/run -a output/test.S -c /path/to/reference-chip/emulator/emulator'
|
|
|
|
*** Currently, due to the limiation of scala process library, you cannot
|
|
torture the RTL simulator ***
|
|
# Generate a random test and diff the signatures of ISA and RTL simulators:
|
|
# % ./sbt 'testrun/run -r /path/to/reference-chip/vlsi/build/vcs-sim-rtl/simv'
|
|
|
|
Run tests for 30 minutes, email hcook when done, and save failures to dir:
|
|
% ./sbt 'overnight/run -m 30 -e hcook@eecs.berkeley.edu -p dir'
|
|
|
|
---------------------------------------------------------------------------
|
|
Installing
|
|
---------------------------------------------------------------------------
|
|
|
|
% git submodule update --init
|
|
|
|
|
|
---------------------------------------------------------------------------
|
|
Overnight Overview
|
|
---------------------------------------------------------------------------
|
|
|
|
This framework utilizes both the test runner and test generator to perform
|
|
a long terms serach for failing test cases. It takes the following command
|
|
line arguments:
|
|
|
|
Usage: overnight/run [options]
|
|
-C <file> | --config <file>
|
|
config file
|
|
-p <dir> | --permdir <dir>
|
|
dir to store failing tests
|
|
-c <file> | --csim <file>
|
|
C simulator
|
|
-r <file> | --rtlsim <file>
|
|
RTL simulator
|
|
-e <address> | --email <address>
|
|
email to report to
|
|
-t <count> | --threshold <count>
|
|
number of failures to trigger email
|
|
-m <int> | --minutes <int>
|
|
number of minutes to run tests
|
|
|
|
You can only generate tests with one instruction mix at a time (based on
|
|
the setting in the config file). It doesn't matter what simulator you use
|
|
with the -r and -c flags, they just determines the name used to describe
|
|
whose diff failed.
|
|
|
|
---------------------------------------------------------------------------
|
|
Testrun Overview
|
|
---------------------------------------------------------------------------
|
|
|
|
This utility compares the signatures generated by passing the -testsig flag
|
|
to the specified simulators. If it encounters a difference, it subdivides
|
|
the test into many subtests and searches for which exact program segment
|
|
reveals the failure. It takes the following command line arguments:
|
|
|
|
Usage: testrun/run [options]
|
|
-C <file> | --config <file>
|
|
config file
|
|
-a <file> | --asm <file>
|
|
input ASM file
|
|
-c <file> | --csim <file>
|
|
C simulator
|
|
-r <file> | --rtlsim <file>
|
|
RTL simulator
|
|
-s <boolean> | --seek <boolean>
|
|
Seek for failing pseg
|
|
-d <boolean> | --dump <boolean>
|
|
Dump mismatched signatures
|
|
|
|
If you don't specify a asm file, a random one will be generated for you.
|
|
You can only generate tests with one instruction mix at a time (based on
|
|
the setting in the config file). It doesn't matter what simulator you use
|
|
with the -r and -c flags, they just determines the name used to describe
|
|
whose diff failed. By default, a failed diff will result in the subtest
|
|
sweep occuring, but this search can be diasbled. Note that the pseg ID
|
|
reported is actually the pseg following the pseg containing the error.
|
|
You can optionally dump mistmatched signatures to the dir containing the
|
|
asm file under test.
|
|
|
|
---------------------------------------------------------------------------
|
|
Generator Overview
|
|
---------------------------------------------------------------------------
|
|
|
|
To generate a random test, the torture test generator randomly generates
|
|
many test sequences from a set of test sequences that are written by hand,
|
|
performs a random register allocation for all test sequences, and finally
|
|
randomly interleaves instructions from these test sequences. To extend the
|
|
set of tests or coverage, the programmer needs to write new test sequences.
|
|
It takes the following command line arguments:
|
|
|
|
Usage: generator/run [options]
|
|
-o <filename> | --output <filename>
|
|
output filename
|
|
-C <file> | --config <file>
|
|
config file
|
|
|
|
The following sections describe adding new functionality to the generator.
|
|
|
|
---------------------------------------------------------------------------
|
|
Test sequence example
|
|
---------------------------------------------------------------------------
|
|
|
|
Before we talk about how to write a test sequence, let's look at a very
|
|
simple example. The following example is a test sequence, which emits an
|
|
add instruction.
|
|
|
|
class SeqADD extends Seq
|
|
{
|
|
val src1 = reg_read_any()
|
|
val src2 = reg_read_any()
|
|
val dest = reg_write(src1, src2)
|
|
insts += ADD(dest, src1, src2)
|
|
}
|
|
|
|
As I hinted in the overview that the test generator will do register
|
|
allocation you don't write a string of instructions with architectural
|
|
registers. You request a virtual registers (i.e., registers that are yet
|
|
tied down to architectural registers) when you need them, save them in
|
|
scala values, and use them when you need to (e.g., in an instruction).
|
|
|
|
---------------------------------------------------------------------------
|
|
Types of virtual registers
|
|
---------------------------------------------------------------------------
|
|
|
|
- Hidden (position dependent registers): Registers that will have
|
|
different values when the code is positioned at a different address. An
|
|
example is registers that hold addresses. Registers that are hidden should
|
|
be excluded from the output signature.
|
|
|
|
- Visible (position independent registers): Registers that are not hidden,
|
|
therefore will have the same values when the code is positioned at
|
|
a different address. These registers should be included as part of the
|
|
output signature.
|
|
|
|
---------------------------------------------------------------------------
|
|
How to write a sequence
|
|
---------------------------------------------------------------------------
|
|
|
|
Use the following functions to request a register, and generate a string of
|
|
instructions (look at Inst.scala to see what instructions are available)
|
|
that uses these virtual registers, and add them to the insts array.
|
|
|
|
- reg_read_zero(): returns register x0
|
|
- reg_read_any(): returns any type of register (hidden or visible)
|
|
- reg_read_visible(): returns a visible register
|
|
- reg_write_ra(): returns register ra for write
|
|
- reg_write_visible(): returns a visible register for write
|
|
- reg_write_hidden(): returns a hidden register for write
|
|
- reg_write(regs: Reg*): returns a register that matches the type of regs
|
|
(if any reg in regs are hidden, the output type is hidden)
|
|
|
|
Note that the torture test framework is written in scala, you can use any
|
|
scala functionality to generate instructions. Look at SeqALU.scala,
|
|
SeqMem.scala, and SeqBranch.scala to get inspired.
|
|
|
|
---------------------------------------------------------------------------
|
|
Future TODO
|
|
---------------------------------------------------------------------------
|
|
|
|
- provide support for loops
|
|
|
|
- generate statistics of a test to get a sense of coverage
|
|
+ statistics should include instruction count of each type
|
|
+ statistics should include register usage
|
|
|
|
- complete floating point tests
|
|
+ add floating point memory move tests
|
|
+ improve floating point init randomization
|
|
+ add rounding modes tests
|
|
|
|
- complete vector tests
|
|
+ better randomization
|
|
+ add SeqVOnly: Tests special vf-only instructions
|
|
|
|
- code refactoring
|
|
+ consolidate RegPool logic
|
|
+ detect and suppress unallocatable sequences
|