Benchmarking is a good way to see if one’s code is effective. The various programming solutions can be compared by taking the time it takes for the interpreter to compute the function. Benchmarking normally works in the way that the system time is taken before performing the function, then the function is applied, and finally the time is taken again, subsequently subtracted by the first time.
Here is the benchmark function of SuperCollider:
bench { arg print = true;
var dt;
var t0 = Main.elapsedTime;
this.value;
dt = Main.elapsedTime - t0;
if (print) { Post << "time to run: " << dt << " seconds.\n"; }
^dt
}
I thought it would be good to have such a function available when developing in Impromptu. By putting the function in a .scm file in the Application Support/Impromptu folder, it would be read on startup and be accessible at all times. The idea to create this benchmark function came when pondering what is the most effective way of populating lists (see last post). So before we start, here are the functions that I will benchmark:
; descending list
(define populateD
(lambda (n)
(if (= 0 n)
'()
(cons n (populateD (- n 1))))))
(populateD 52)
; ascending list
(define populateA
(lambda (n end)
(if (= n end)
'()
(cons n (populateA (+ n 1) end) ))))
(populateA 0 22)
; ascending list using iteration (dotimes)
(define populateI
(lambda (n)
(define counter 0)
(define ls '())
(dotimes (i n)
(set! counter (+ counter 1))
(set! ls (cons (- n i) ls)))
ls))
(populateI 22)
; ascending list using an "inner recursion" through let
(define populateR
(lambda (n)
(let loop ((cnt 0))
(if (> cnt n)
'()
(cons cnt (loop (+ cnt 1) ))))))
(populateR 33)
; and finally ascending list using recursion using apply
(define populateRA
(lambda (n . args)
(if (< n 0)
args
(apply populateRA (- n 1) (cons n args)))))
(populateRA 22)
I started my experiment in creating benchmarking code by replicating the SuperCollider function:
(begin
(define a (now))
(populateA 0 4222)
(define b (now))
(print "benchmark time : " (- b a)))
This works, but the problem is that (now) gives times in samples since Impromptu started, and it does so in sample blocks, resulting in benchmark times of 0, 256, 512, 768, etc. (i.e., with block size of 256 samples). Using (clock) solves this issue (although I didn’t know of clock until it was pointed out on the mailing list. The Impromptu Function List document does not have any reference to clock). Next step was to use clock and put it into a function:
(define bench2
(lambda()
(begin
(define t1 (clock))
(populateA 0 22)
(define t2 (clock))
(print "benchmark time : " (- t2 t1)))))
(bench2)
All good. But obviously a the function to be benchmarked should not be hardcoded into the bench function: it should be passed to it:
(define bench
(lambda (func)
(begin
(define t1 (clock))
func
(define t2 (clock))
(print "benchmark time : " (- t2 t1)))))
(bench (populateA 0 22))
Here I stumbled into an interesting difference between SuperCollider and Impromptu, elegantly explained by Andrew Sorensen in his reply to my mail: “What you’re striking is the difference between call-by-value and call-by-name (often called lazy evaluation – as per Haskell). Scheme is call-by-value, which in brief means that it evaluates any argument expressions before applying their values to the given function. When you try to apply the function bench i.e. (bench (reverse (populateR 220))) the expresion (reverse (populateR 220)) first gets evaluated and it’s result (i.e. a list) is then sent as an argument to bench.”
Lazy evaluation is probably not the right description of what is happening in the SC bench function. There the function itself is passed and evaluated in the right place. (between the two time samples).
Anyway, Thorin suggested the following bench function:
(define profile
(lambda (iternum func . args)
(let ((*time* (clock)))
(dotimes (i iternum)
(apply func args))
(set! *time* (- (clock) *time*))
(print "benchmark time: " *time*))))
(profile 100 populateA 0 22)
and here the function is evaluated x amount of times and finally printing the time it takes for it to perform. This works fine in principle, but as Thorin acknowledges, there is a certain overhead in using (dotimes) and (apply) x number of times. This can be seen if we compare the two benchmarks of bench2 and profile:
(bench2) ; "benchmark time : " 0.000076
(profile 1 populateA 0 22) ; "benchmark time: " 0.000867
This is therefore not suitable if I want to compare the performance of Impromptu versus other languages. For that I need the most simple benchmark function possible, ideally mirroring the SuperCollider function. Andrew suggested the following:
(define bench
(lambda (func . args)
(let ((t (clock))
(res (apply func args)))
(print "benchmark time :" (- (clock) t))
res)))
which works well, except it is still using (apply) and it won’t work with nested function calls, such as (reverse (populateA 22)). This note from Andrew explains what happens well: “the populateR expression (an atom is still an expression) is still evaluated, but it is evaluating the symbol which is bound to a function – whose evaluation returns the bound function (i.e. does not apply the function). If we put (bench (populateR 10)) we are instead evaluating the expression (populateR 10) and then passing the result of evaluating that expression to bench. (of course 10 also gets evaluated before being applied to popluateR but numbers evaluate to themselves – as do strings and characters).”
Andrew then suggests making a macro (something I will study next) for this task:
(define-macro (benchm expr)
`(let ((t (clock))
(res ,expr))
(print "benchmark time:" (- (clock) t))
res))
(benchm (reverse (populateR 220)))
(benchm (populateI 220))
(map (lambda (i) (print 'myprint: i) i) (benchm (populateR 10)))
Voila! We’ve got our benchmark function, so let’s do some testing of the populateList functions above:
(benchm (populateD 220)) ; "benchmark time:" 0.000321
(benchm (populateA 0 220)) ; "benchmark time:" 0.000350
(benchm (populateI 220)) ; "benchmark time:" 0.001155
(benchm (populateR 220)) ; "benchmark time:" 0.000332
(benchm (populateRA 220)) ; "benchmark time:" 0.000377
This is obviously not very scientific but gives some indication, although the resulting times vary quite a lot. It is clear that the iterative function that uses (dotimes) is twice slower than the others. In general the (populateR) function is the fastest one, probably because it is not using (apply) as the (populateRA) function.
Finally, Impromptu has some inbuilt functions for list operations so I tried to benchmark one that does what populateR does:
(benchm (make-list-with-proc 22
(let ((n 0))
(lambda () (set! n (+ n 1)) n)))) ; "benchmark time:" 0.000556
Interestingly, this is a bit slower than the methods above. I checked if it is a macro, and it’s not. Finally I found it in the Impromptu.scm file in the Impromptu.app Resources.
Here is how that function looks:
(define (make-list-with-proc lth func)
(if (< lth 1)
'()
(let loop ((i 0)
(lst '()))
(if (>= i lth)
(reverse lst)
(loop (+ i 1) (cons (func i) lst))))))
It is doing the loop, and it is evaluating the function that is passed to it as well. I think this explains why it is slightly slower.
Anyway, performing a bit of testing of SC, Python and Impromptu looks like this:
bench{
a = [];
22.do({arg i; a = a.add(i) });
}
time to run: 6.3839999995707e-05 seconds.
0.000063839 seconds
Python code might look like this:
import time
def bench() :
b = time.clock()
a = []
for i in range(22):
a.append(i)
c = time.clock()
print "benchmark time : " + str(c-b)
bench()
which returns:
benchmark time : 6.2e-05
i.e. 0.000062
So Impromptu seems to be around 10 times slower than Python and SC for these types of operations. Obviously this comparison can be flawed as I am no benchmarking specialist, but this is what meets the eye at first glance.
Furthermore, with an array of 22000 it takes Python 0.0062 seconds and SuperCollider 0.00972 seconds. In Impromptu, the benchmark time using make-list-with-proc takes 0.1139 seconds, and using the populateR method 0.1197 seconds.
Trying to populate an array of, say, 130000 items is problematic in Impromptu. It crashes the language. I’m not sure why, but before it crashes it posts lots of these messages:
:ERROR: position:(0) in function “(null)”
illegal function
Trace:
These problems that I’m stumbling into are hardly problems for music making in Impromptu. If one schedules musical events ahead of time, functions should be able to finish before the event, and one is not likely to make an array of 130000 items, except if wanting to work on sample operations. I am curious about these “shortcomings” of Impromptu though.
