This code seems to be not working:
layout.booleanTool.boolOnLayer(17,88,17, "A-B", nx=0, ny=0, borderMerge=0);
Can you check if there is something wrong ?
nx=0, ny=0 and borderMerge=0 are default parameter. If you leave the parameter away, the listed value is assumed. So the command
layout.booleanTool.boolOnLayer(17,88,18, "A-B");
is identical with
layout.booleanTool.boolOnLayer(17,88,18, "A-B", 0, 0, 0);
or with
layout.booleanTool.boolOnLayer(17,88,18, "A-B", 0);
That is identical in C++-Macros or LayoutScript for Ruby.