You can use Testdriver with different types of software. Here, we will explain how to import and run third-party benchmarks and other software that can be launched from the command line.
To run third-party software with Testdriver you will need to create a user-defined batch file. This batch script acts as the entry point, and of course, it can execute scripts written with Powershell or other languages, if you prefer.
To create a custom benchmark package and import it to Testdriver, follow these steps.
Please ensure the scripts are using UTF-8 or they may not be parsed correctly.
- Write at least one Windows batch script which runs the benchmarks, handles the result output, and returns it to Testdriver in one of two defined ways. The script can write the results to a file, in which case multiple metrics can be returned at once, or it can simply return an exit code to the command line when the script terminates.
- Bundle one or more of these .bat scripts with any desired filenames, along with all the other required files (such as benchmark executables and resources), into a .zip archive. Batch files must be placed into the archive’s root folder.
If the script returns only an exit code, use Import Resultless Script to import the .zip package.
If the script creates a result file, use the Import Benchmark Script button to import the .zip package.
The result file must adhere to the following specification for Testdriver to parse it. The location of the file will be X:\Testdriver\tempresult\raw_result.json, where X is the letter of the drive where the Testdriver Server is installed.
The format of the JSON file is as follows:
{
"workload": {
"name": "Battery"
},
"outputs": [{
"resultType": "Remaining charge (percent)",
"value": 99
}]
}A workload object containing a name field is used to identify the test. The outputs array can contain any number of objects which have a resultType property — essentially the name of the score — and a value property, which must be a number.
Running third-party benchmarks
Testdriver supports running third-party benchmarks and can run almost anything that can be run with a script. The example below uses a .bat file to run all Cinebench tests via Powershell, and collects the results which are sent to the Tetsdriver server and stored in the Testdriver database.
When scheduling third-party benchmark runs, the command line to be executed must always be specified, even if the imported script consists of a single executable. For example, to schedule this example, the command line command run_all.bat needs to be explicitly specified on the Run Control screen.
run_all.bat
"CINEBENCH Windows 64 Bit.exe" -cb_all "CINEBENCH Windows 64 Bit.exe" -cb_all > result_all.txt powershell -ExecutionPolicy Bypass -file ./cinebenchToTd.ps1 all
cinebenchToTd.ps1
[CmdletBinding()]
Param(
[Parameter(Mandatory=$True,Position=1)]
[string]$Benchmark
)
$data = get-content "./result_$Benchmark.txt" -raw
$cinebenchResultTypes = @{
"opengl" = @("Shading (OpenGL)", "Shading (OpenGL) Reference Match")
"cpu1" = @("Rendering (Single CPU)")
"cpux" = @("Rendering (Multiple CPU)")
"all" = @("Shading (OpenGL)", "Shading (OpenGL) Reference Match", "Rendering (Single CPU)", "Rendering (Multiple CPU)")
}
$dict = @{}
$b = $data -split "`r`n"
foreach ($line in $b)
{
$pair = $line -split ":"
if ($cinebenchResultTypes[$Benchmark].Contains($pair[0].Trim())) {
$dict.add($pair[0].Trim(), $pair[1].Trim())
}
}
$results = @()
foreach ($item in $dict.Keys) {
$results= $results + @{
"resultType" = $item
"value" = [double]::Parse($dict[$item] -replace "[^0-9|.]", [CultureInfo]::InvariantCulture)
}
}
If(!(Test-Path -path "C:\Testdriver\tempresult\"))
{
#if it does not create it
New-Item -ItemType Directory -Force -Path "C:\Testdriver\tempresult\"
}
$json = @{
"workload"= @{
"name"= "Battery"
}
"outputs"= $results
} | ConvertTo-JSON -Depth 5
[IO.File]::WriteAllLines("C:\Testdriver\tempresult\raw_result.json", $json)