Latest Version: 0.2.x

What is Testarium?

Testarium is a research tool to perform experiments and store results as in the repository (git or mercurial). It implements scientific template of experiments and uses numpy, colorama, flask, d3, jquery-ui and angular to provide powerful shell and beautiful presentation of your experimental data and scores.

It can be helpful for science researchers and their bosses to monitor the work progress and to optimize parameters.

Setup

$ pip install testarium

Quick start

project/config/config.json
{
	"a" : 0.5
}
project/example.py
#!/usr/bin/env python
import random
import testarium
import testarium.score.fafr


@testarium.experiment.set_run
def my_run(commit):
    # prepare
    d = commit.dir
    c = commit.config

    # run experiment
    pos = open(d + '/pos.txt', 'w')
    neg = open(d + '/neg.txt', 'w')
    [pos.write(str(random.random() * c['a']) + '\n') for _ in xrange(100)]
    [neg.write(str(random.random() * c['a']) + '\n') for _ in xrange(100)]
    return 0


@testarium.experiment.set_score
def my_score(commit):
    d = testarium.score.fafr.Score(commit.dir)
    d['test.param'] = 1
    return d


@testarium.testarium.set_print
def my_print(commit):
    score = '%0.2f' % (commit.desc['score'] * 100.0)

    return ['name', 'a', 'test', 'score'], \
           [commit.name, commit.config['a'], commit.desc['test.param'], score]


@testarium.testarium.set_compare
def my_compare(self, other):
    if self._init:
        if self.desc['score'] > other.desc['score']:
            return -1
        elif self.desc['score'] < other.desc['score']:
            return 1
        else:
            return 0
    else:
        return -1


if __name__ == '__main__':
    testarium.testarium.best_score_is_min()
    testarium.main()
$ python example.py run
t> New commit: 20141220.162326
t> 20141220.162326 > branch: default > score: 1.06 > time: 0.0

Testarium paradigm

Typical scientific template of the experiment setup:

Main testarium unit is a commit. The commit consists of two parts: config and result scores & data (experiment description, desc). Each experiment run = new testarium commit.

Contribute

Found a bug? Have a good idea for improving Testarium? Head over to Testarium's github page and create a new ticket or fork. You can also add issues and feature requests directly to the issue tracker. Fork me on GitHub