As Simple as Possible but no Simpler

[rocky]

A number of time people report bugs with extremely complicated examples. I often get lost in before I even get to the problem.

Here is a recent example:

Solve[Q[x,y]==1/6(x+y+1/x+1/y+y/x+x/y)(Q[x,y]-Q[x,0]-Q[0,y]+Q[0,0])+1/2(x+y)Q[0,0]+(1/3 x+1/6(y+1/x+y/x+1))(Q[x,0]-Q[0,0]) + (1/3 y+1/6(x+1/y+x/y+1))(Q[0,y]-Q[0,0]),Q[x,y]];

gives:

-\frac{y \left(-1+x\right) \left(1+y\right)}{x+x^2+y-6 x y+x^2 y+y^2+x y^2}

(I added markup above where the bug reporter didn't bother. At least for me, laziness in bug reporting is not something that motivates me to take the trouble to get involved. I have a lot of other things I could be doing.)

Lest you think that an isolated example, here is another:

A=7.17;
e=0.9;
Clw=0.71;
Xcg=0.591;
Xca=0.50075;
N0=0.94141;
at=4.15;
aw=4.02;
St=4.06;
Sw=21.3;
w=2200;
Lt=6;
cam=2.003;
nt=0.9;
Kyy=1.75;
deda=0.355568;
Cdf=0.048;
dh=0.065;
ce=1.05;
ct=0.85;
Vc=400/3.6;
Se=4.06;
me=42/2;
xe=(0.65-.0.65-0.3);
ke=0.7;
ro=0.00162214.54/0.0283/9.81;
CTiempo = w/(9.81roVcSw)
Clalfa=aw
Cd=Cdf+(ClwClw)/(PiAe)
Cdalfa = 2ClwClalfa/(PiAe)
Cmalfa = Clalfa(Xcg-N0)
Xa =Xcg - Xca
mu = (w/9.81)/(roSwcam)
Vred=(Lt-Xa)St/(camSw)
Cmdalfa=-atVrednt1/mu(Lt-Xa)deda/cam
CmdTita = -1.1atVrednt*(Lt-Xa)/(cam*mu)

This is the system:

DL={{Cd+lambda, (Cdalfa-Clw)/2, Clw/2},{Clw, Clalfa/2+lambda, -lambda},{0,Cmalfa+Cmdalfalambda,CmdTitalambda-2lambda^2Kyy^2/(cam^2*mu)}};

The same kind of thing isn't limited to just bugs in the answers Mathics produces. We can find it in performance bugs as well.

For example here is a performance bug related to a probably recent regression in our code for 4.1.0.

Do[F[a,a,a,a,a,a,a,a,a,a,a];,{1000}]

The thing I'd like to focus on is the "1000" the number of times this Do is run. Clearly numbers like 1 and 2 are too small, but Is 1000 really necessary?

It turns out that our performance bug has something to do with the function from_python() getting called much more than before:

         683886 function calls (665731 primitive calls) in 0.483 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
...
    15018    0.044    0.000    0.047    0.000 atoms.py:1062(from_python)
vs 4.0.0:

versus previously:

         602313 function calls (584117 primitive calls) in 0.359 seconds

     2016    0.007    0.000    0.013    0.000 expression.py:95(from_python)

Here, I find that we can see something jump out even in as few as 10 iterations of the loop. If you want things to stand out more, then at 50, the difference is clear enough.

Why this matters is that if you are automating benchmarking and running this many times, adding delays is not good. The slower the tests are, the less likely people will run them.

In conclusion, I will say that when you encounter a bug it is likely that it won't be in its simplest form - it takes work to get it to figure out how to narrow a bug so it is.

Similarly when initially coming up with benchmark tests, 1000 might be something to start out with. However after a problem is identified, that is when you probably want to narrow things.