From rseba at dit.unitn.it Tue Jun 6 09:04:36 2006 From: rseba at dit.unitn.it (Roberto Sebastiani) Date: Tue Jun 6 09:04:43 2006 Subject: [SMTCOMP] IMPORTANT: the infinite-precision issue Message-ID: Dear colleagues, there is an important question about SMT-COMP which apparently nobody asked so far: do we explicitly require infinite precision from SMT solvers on arithmetic-related theories? (I.e., RDL/IDL/LAR/LAI and combinations.) On the one hand, infinite precision prevents an SMT tool from giving wrong answers on some problems. On the other hand, using infinite precision significantly slows down the performances (around a 3 factor in our experience). Thus, we believe that, in order to guarantee a fair comparison, this issue should be decided a priori and clearly stated in the competition rules. In particular: 1 if we require infinite precision, then the benchmarks of all categories should contain a significant amount of problems (even very easy ones) which cause wrong answers if dealt with non-infinite arithmetic; 2 if we do not require infinite precision, then this must be decided a priori and clearly stated in the rules, so that tools using infinite precision can decide with a reasonable advance to use standard arithmetic instead. Roberto -- ------------------------------------------------------------------ Prof. ROBERTO SEBASTIANI Dip. Informatica e Telecomunicazioni Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba ------------------------------------------------------------------ From demoura at csl.sri.com Tue Jun 6 15:45:52 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Tue Jun 6 15:45:55 2006 Subject: [SMTCOMP] IMPORTANT: the infinite-precision issue In-Reply-To: References: Message-ID: Hi Roberto, This is a good point. > there is an important question about SMT-COMP which apparently > nobody asked so far: do we explicitly require infinite precision > from SMT solvers on arithmetic-related theories? (I.e., RDL/IDL/LAR/ > LAI and combinations.) I believe one needs to use multi precision arithmetic for solving some problems in LRA and LIA, but I agree with you that the benchmarks in RDL and IDL do not require infinite precision. I'm not sure if we should force/require people to use multi precision arithmetic. I think it is perfectly ok to use fixed precision, and return "unknown" when an overflow/underflow is detected. I also believe it is perfectly ok to enter an incomplete solver, and return "unknown" for formulas you know your solver is incomplete. For example, Aaron Stump did that last year. CVC is not complete for integers, and he returned "unknown" for potential "sat" instances. Anyway, I really interested in knowing what the rest of the SMT community thinks about this issue. > On the one hand, infinite precision prevents an SMT tool from > giving wrong answers on some problems. On the other hand, using > infinite precision significantly slows down the performances > (around a 3 factor in our experience). In my experiments, I noticed a 1.4 factor. I'm using GMP (www.swox.com/gmp/) for infinite precision. I guess you are using it too. :-) We found that memory management can be expensive with GMP, and a good trick is to create a pool of GMP numbers and reuse them for intermediate computations. Cheers, Leonardo. From tinelli at cs.uiowa.edu Wed Jun 7 07:17:03 2006 From: tinelli at cs.uiowa.edu (Cesare Tinelli) Date: Wed Jun 7 07:17:33 2006 Subject: [SMTCOMP] IMPORTANT: the infinite-precision issue In-Reply-To: References: Message-ID: <4486DFDF.9050303@cs.uiowa.edu> Hi all, Leonardo de Moura wrote: > Hi Roberto, > > This is a good point. > >> there is an important question about SMT-COMP which apparently nobody >> asked so far: do we explicitly require infinite precision from SMT >> solvers on arithmetic-related theories? (I.e., RDL/IDL/LAR/LAI and >> combinations.) > > I believe one needs to use multi precision arithmetic for solving some > problems in > LRA and LIA, > It looks like it is needed in some problems. According to Domagoj Babic who has looked at the QF_LIA problems in depth, at least one of the *64*.msat.smt problems in http://goedel.cs.uiowa.edu/smtlib/smtlib/benchmarks/QF_LIA/CIRC use numerals that *will* go out or range on 32bit machines. > but I agree with you that the benchmarks in RDL and IDL do not > require infinite precision. > > I'm not sure if we should force/require people to use multi precision > arithmetic. > I think it is perfectly ok to use fixed precision, and return "unknown" > when an overflow/underflow is detected. > > I also believe it is perfectly ok to enter an incomplete solver, and return > "unknown" for formulas you know your solver is incomplete. > For example, Aaron Stump did that last year. CVC is not complete > for integers, and he returned "unknown" for potential "sat" instances. > > Anyway, I really interested in knowing what the rest of the SMT community > thinks about this issue. > Domagoj had this suggestion: how about annotating each benchmark with an indication of what kind of precision it needs? This could be easily done in our open format by adding an additional field. Domagoj was suggesting something of the form :max_range {infty | u64 | s32 | ... } specifying the minimum range required (u64 - 64bit unsigned, s32 - 32bit signed, ...) to solve the benchmark correctly. The problem of course would be to figure out the smallest safest range for the existing benchmarks and future ones when it is not explicitly provided by the benchmarks' creator. What do you think? Cesare >> On the one hand, infinite precision prevents an SMT tool from >> giving wrong answers on some problems. On the other hand, using >> infinite precision significantly slows down the performances (around a >> 3 factor in our experience). > > In my experiments, I noticed a 1.4 factor. I'm using GMP > (www.swox.com/gmp/) > for infinite precision. I guess you are using it too. :-) > We found that memory management can be expensive with GMP, and a good trick > is to create a pool of GMP numbers and reuse them for intermediate > computations. > > Cheers, > Leonardo. > > > From pocmatos at gmail.com Wed Jun 7 07:44:25 2006 From: pocmatos at gmail.com (Paulo J. Matos) Date: Wed Jun 7 07:44:31 2006 Subject: [SMTCOMP] IMPORTANT: the infinite-precision issue In-Reply-To: <4486DFDF.9050303@cs.uiowa.edu> References: <4486DFDF.9050303@cs.uiowa.edu> Message-ID: <11b141710606070744x6541980dja763f9aa22c66006@mail.gmail.com> Hi all, On 07/06/06, Cesare Tinelli wrote: > Domagoj had this suggestion: how about annotating each benchmark with an > indication of what kind of precision it needs? > This could be easily done in our open format by adding an additional > field. Domagoj was suggesting something of the form > > :max_range {infty | u64 | s32 | ... } This will not solve Sebastiani's the problem, right? Since, even if you annotate it, you have to build your solver against GMP in case max_range is infty. Unless, you can decide during runtime which type of precision to use, this doesn't solve it. Obviously, you can define an interface and two math engines, one based on processor 'precision' and other based on 'GMP' and then load one of them once you know which kind of precision to use but it's not a simple solution. > > specifying the minimum range required (u64 - 64bit unsigned, s32 - 32bit > signed, ...) to solve the benchmark correctly. > You mean the maximum precision? > The problem of course would be to figure out the smallest safest range > for the existing benchmarks and future ones when it is not explicitly > provided by the benchmarks' creator. > > What do you think? > > If you decide to go through this path then, every non-unspecified benchmark against max_range should be considered to be infty, probably. That is if we prefer correctness and completeness to efficiency. On the other hand, I don't think removing benchmarks with infty precision requirements is a solution since there may be solvers using languages, which by default have infty precision and are then already paying a price. Removing these benchmarks are unfair to them. As I see it, the solver should return unknown if it cannot solve benchmarks with infty precision and it detects some overflow, as it was already mentioned by Leonardo. By doing this, you'll leave the weight of the decision on the authors which should decide what to do. There are always solutions which authors might take, like trying to solve using limited precision and shifting to a GMP interface during runtime if overflow is detected. Solver is more complex but solves the issue. Cheers, Paulo Matos > > Cesare > > > >> On the one hand, infinite precision prevents an SMT tool from > >> giving wrong answers on some problems. On the other hand, using > >> infinite precision significantly slows down the performances (around a > >> 3 factor in our experience). > > > > In my experiments, I noticed a 1.4 factor. I'm using GMP > > (www.swox.com/gmp/) > > for infinite precision. I guess you are using it too. :-) > > We found that memory management can be expensive with GMP, and a good trick > > is to create a pool of GMP numbers and reuse them for intermediate > > computations. > > > > Cheers, > > Leonardo. > > > > > > > > -- Paulo Jorge Matos - pocm at sat inesc-id pt Web: http://sat.inesc-id.pt/~pocm Computer and Software Engineering INESC-ID - SAT Group From stump at cse.wustl.edu Wed Jun 7 08:25:00 2006 From: stump at cse.wustl.edu (Aaron Stump) Date: Wed Jun 7 08:25:15 2006 Subject: [SMTCOMP] IMPORTANT: the infinite-precision issue In-Reply-To: <11b141710606070744x6541980dja763f9aa22c66006@mail.gmail.com> References: <4486DFDF.9050303@cs.uiowa.edu> <11b141710606070744x6541980dja763f9aa22c66006@mail.gmail.com> Message-ID: <25630.1149693900@cse.wustl.edu> [ responding to Cesare: ] > Domagoj had this suggestion: how about annotating each benchmark with > an indication of what kind of precision it needs? > This could be easily done in our open format by adding an additional > field. Domagoj was suggesting something of the form > > :max_range {infty | u64 | s32 | ... } > > specifying the minimum range required (u64 - 64bit unsigned, s32 - > 32bit signed, ...) to solve the benchmark correctly. > > The problem of course would be to figure out the smallest safest range > for the existing benchmarks and future ones when it is not explicitly > provided by the benchmarks' creator. I think this sounds like a good idea. Obtaining a rough estimate of the smallest safe annotation could maybe be done (by someone with their fingers in the code of a solver built with a bignum package) by instrumenting the solver to track how large the integers used during solving are. A more mathematical definition of the smallest safe annotation might be challenging to come up with, and challenging to compute (is it the annotation needed for a run of a particular published algorithm? the one needed for the shortest proof in some theory?). Aaron From rseba at dit.unitn.it Thu Jun 8 01:17:35 2006 From: rseba at dit.unitn.it (Roberto Sebastiani) Date: Thu Jun 8 01:17:45 2006 Subject: [SMTCOMP] IMPORTANT: the infinite-precision issue In-Reply-To: References: Message-ID: Hi all. I think the discussion has gone a little far from the original issue I raised. Thus, let me do one step back. (First, Leonardo, notice that we're perfectly at ease with the idea of incomplete solvers, but this was not intended to be the issue of my email.) My issue was: we WANT CLEAR RULES ABOUT INFINITE PRECISION. That is, for each category, either: 1) we state that infinite precision is required, and we add a significant amount of "critical" problems (which may cause wrong solution if plain finite precision is used), or 2) we state that infinite precision is *not* required, and we include no such critical problem. In the first case, each tool is guaranteed againts unfair comparisions with non-infinite-precision tools. In the second case, one can safely release infinite precision to that to have a fair competition with non-infinite-precision tools. Notice that we're not pushing for either direction: we're simply pushing for clear rules. Roberto PS: as far as Cesare's proposal is concerned, I agree with the arguments of Paulo Matos. Moreover, from a practical perspective, I think this would add a lot of extra workload in estabilishing the ranges and/or ask the authors. (BTW: who guarantees that the authors of the problems are right?). >From a more principled perspective, I don't like the idea of customizing tools for single instances, or of adopting tricks like "if infty run mathsat_infty else if 32 run mathsat_32 else run mathsat_64". On Tue, 6 Jun 2006, Leonardo de Moura wrote: > Hi Roberto, > > This is a good point. > >> there is an important question about SMT-COMP which apparently nobody asked >> so far: do we explicitly require infinite precision from SMT solvers on >> arithmetic-related theories? (I.e., RDL/IDL/LAR/LAI and combinations.) > > I believe one needs to use multi precision arithmetic for solving some > problems in > LRA and LIA, but I agree with you that the benchmarks in RDL and IDL do not > require infinite precision. > > I'm not sure if we should force/require people to use multi precision > arithmetic. > I think it is perfectly ok to use fixed precision, and return "unknown" > when an overflow/underflow is detected. > > I also believe it is perfectly ok to enter an incomplete solver, and return > "unknown" for formulas you know your solver is incomplete. > For example, Aaron Stump did that last year. CVC is not complete > for integers, and he returned "unknown" for potential "sat" instances. > > Anyway, I really interested in knowing what the rest of the SMT community > thinks about this issue. > >> On the one hand, infinite precision prevents an SMT tool from >> giving wrong answers on some problems. On the other hand, using infinite >> precision significantly slows down the performances (around a 3 factor in >> our experience). > > In my experiments, I noticed a 1.4 factor. I'm using GMP (www.swox.com/gmp/) > for infinite precision. I guess you are using it too. :-) > We found that memory management can be expensive with GMP, and a good trick > is to create a pool of GMP numbers and reuse them for intermediate > computations. > > Cheers, > Leonardo. > > -- ------------------------------------------------------------------ Prof. ROBERTO SEBASTIANI Dip. Informatica e Telecomunicazioni Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba ------------------------------------------------------------------ From demoura at csl.sri.com Fri Jun 9 10:24:32 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Fri Jun 9 10:24:37 2006 Subject: [SMTCOMP] new benchmarks Message-ID: Hi, Hyondeuk (Univ. Colorado at Boulder) submitted several benchmarks based on the job shop scheduling problem. They can be downloaded at: http://www.csl.sri.com/users/demoura/smt-comp/benchmarks.shtml We also added a link to last year QF_RDL benchmarks. Leonardo From demoura at csl.sri.com Sat Jun 10 10:54:10 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Sat Jun 10 10:54:12 2006 Subject: [SMTCOMP] IMPORTANT: the infinite-precision issue In-Reply-To: References: Message-ID: Aaron, Clark and I have the following position on the subject: 1) The theories are exactly as stated. This means solvers should support multi-precision arithmetic or they risk being incomplete, or even unsound if they do not detect under and overflows. If someone wants to propose a new theory or logic that limits the size of integers, they are free to do so. 2) We collect the best benchmarks we can based on real problems. We don't think we should fill our benchmark pool with artificial problems in order to bias the results towards certain kinds of solvers. Moreover, we already have several benchmarks in QF_LIA (CIRC) and QF_LRA (clock_synchro) that require multi-precision arithmetic. At least in the algorithm I use, one cannot express intermediate results using an integer or a pair of integers for rationals. 3) We publish the benchmarks and the criteria for choosing the competition benchmarks. Everyone has the same information. And, everybody was welcome to submit benchmarks to the competition. That said, we will enforce that (where applicable) each division contains at least one benchmark that requires bignums, kind of like we did with integers last year. That way, people have some incentive to move to multi-precision arithmetic. So, in the next revision of the benchmark library, we will include one benchmark that requires bignums to the following divisions: QF_IDL, QF_RDL, QF_UFIDL, QF_UFLIA, and QF_AUFLIA. Best wishes, Leonardo for Aaron and Clark, too. On Jun 8, 2006, at 1:17 AM, Roberto Sebastiani wrote: > Hi all. > I think the discussion has gone a little far from the original > issue I raised. > Thus, let me do one step back. > > (First, Leonardo, notice that we're perfectly at ease with the idea > of incomplete solvers, but this was not intended to be the issue of > my email.) > > My issue was: we WANT CLEAR RULES ABOUT INFINITE PRECISION. > > That is, for each category, either: > 1) we state that infinite precision is required, and we add a > significant > amount of "critical" problems (which may cause wrong solution if > plain finite > precision is used), or > 2) we state that infinite precision is *not* required, and we include > no such critical problem. > > In the first case, each tool is guaranteed againts unfair > comparisions with > non-infinite-precision tools. In the second case, one can safely > release infinite precision to that to have a fair competition with > non-infinite-precision tools. > > Notice that we're not pushing for either direction: we're simply > pushing for clear rules. > > Roberto > > PS: as far as Cesare's proposal is concerned, I agree with the > arguments of Paulo Matos. Moreover, from a practical perspective, I > think this would add a lot of extra workload in estabilishing the > ranges and/or ask the authors. > (BTW: who guarantees that the authors of the problems are right?). >> From a more principled perspective, I don't like the idea of >> customizing tools for > single instances, or of adopting tricks like > "if infty run mathsat_infty else if 32 run mathsat_32 else run > mathsat_64". > > > > > > > > > On Tue, 6 Jun 2006, Leonardo de Moura wrote: > >> Hi Roberto, >> >> This is a good point. >> >>> there is an important question about SMT-COMP which apparently >>> nobody asked so far: do we explicitly require infinite precision >>> from SMT solvers on arithmetic-related theories? (I.e., RDL/IDL/ >>> LAR/LAI and combinations.) >> >> I believe one needs to use multi precision arithmetic for solving >> some problems in >> LRA and LIA, but I agree with you that the benchmarks in RDL and >> IDL do not >> require infinite precision. >> >> I'm not sure if we should force/require people to use multi >> precision arithmetic. >> I think it is perfectly ok to use fixed precision, and return >> "unknown" >> when an overflow/underflow is detected. >> >> I also believe it is perfectly ok to enter an incomplete solver, >> and return >> "unknown" for formulas you know your solver is incomplete. >> For example, Aaron Stump did that last year. CVC is not complete >> for integers, and he returned "unknown" for potential "sat" >> instances. >> >> Anyway, I really interested in knowing what the rest of the SMT >> community >> thinks about this issue. >> >>> On the one hand, infinite precision prevents an SMT tool from >>> giving wrong answers on some problems. On the other hand, using >>> infinite precision significantly slows down the performances >>> (around a 3 factor in our experience). >> >> In my experiments, I noticed a 1.4 factor. I'm using GMP >> (www.swox.com/gmp/) >> for infinite precision. I guess you are using it too. :-) >> We found that memory management can be expensive with GMP, and a >> good trick >> is to create a pool of GMP numbers and reuse them for intermediate >> computations. >> >> Cheers, >> Leonardo. >> >> > > -- > ------------------------------------------------------------------ > Prof. ROBERTO SEBASTIANI > Dip. Informatica e Telecomunicazioni > Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 > Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 > roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba > ------------------------------------------------------------------ From alberto.griggio at dit.unitn.it Mon Jun 12 05:05:58 2006 From: alberto.griggio at dit.unitn.it (Alberto Griggio) Date: Mon Jun 12 05:01:55 2006 Subject: [SMTCOMP] smtlib-predabs benchmarks broken? Message-ID: <200606121405.58414.alberto.griggio@dit.unitn.it> Hello all, I'm writing just to signal that the new benchmarks in the smtlib-predabs.tar.gz tarball seem to be malformed: they all contain one extra ')' character at the end of the file... Alberto From demoura at csl.sri.com Mon Jun 12 10:29:40 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Mon Jun 12 10:29:42 2006 Subject: [SMTCOMP] smtlib-predabs benchmarks broken? In-Reply-To: <200606121405.58414.alberto.griggio@dit.unitn.it> References: <200606121405.58414.alberto.griggio@dit.unitn.it> Message-ID: Hi Alberto, Thanks for sending the email. We will fix that. Cheers, Leonardo. On Jun 12, 2006, at 5:05 AM, Alberto Griggio wrote: > Hello all, > I'm writing just to signal that the new benchmarks in the > smtlib-predabs.tar.gz tarball seem to be malformed: they all contain > one extra ')' character at the end of the file... > > Alberto > From tinelli at cs.uiowa.edu Mon Jun 12 15:08:21 2006 From: tinelli at cs.uiowa.edu (Cesare Tinelli) Date: Mon Jun 12 15:08:48 2006 Subject: [SMTCOMP] IMPORTANT: the infinite-precision issue In-Reply-To: References: Message-ID: <448DE5D5.7090805@cs.uiowa.edu> Hi Roberto, I second your request for clear rules. And I leave it to the SMT-COMP organizers to establish what those should be. Here, I just have a couple of comments about your email. Roberto Sebastiani wrote: > Hi all. > I think the discussion has gone a little far from the original issue I > raised. > Thus, let me do one step back. > > (First, Leonardo, notice that we're perfectly at ease with the idea of > incomplete solvers, but this was not intended to be the issue of my email.) > > My issue was: we WANT CLEAR RULES ABOUT INFINITE PRECISION. > > That is, for each category, either: > 1) we state that infinite precision is required, and we add a significant > amount of "critical" problems (which may cause wrong solution if plain > finite > precision is used), or > 2) we state that infinite precision is *not* required, and we include > no such critical problem. > > In the first case, each tool is guaranteed againts unfair comparisions with > non-infinite-precision tools. In the second case, one can safely release > infinite precision to that to have a fair competition with > non-infinite-precision tools. > > Notice that we're not pushing for either direction: we're simply pushing > for clear rules. > > Roberto > > PS: as far as Cesare's proposal is concerned, I agree with the arguments > of Paulo Matos. Moreover, from a practical perspective, I think this > would add a lot of extra workload in estabilishing the ranges and/or ask > the authors. > I do agree with that of course, but I'm not sure how much less work there is in your proposal 1) above. Adding a significant number of critical problems to a division would require the ability to choose them from an even greater pool. Wouldn't identifying or generating problems for this pool require a similar amount of work as well? > (BTW: who guarantees that the authors of the problems are right?). >> From a more principled perspective, I don't like the idea of >> customizing tools for > single instances, or of adopting tricks like > "if infty run mathsat_infty else if 32 run mathsat_32 else run mathsat_64". > Fair enough. But I find it puzzling to hear this from a main MathSAT developer. Isn't MathSAT's layered approach based exactly on the principle of using the right (sub)solver for each specific input (sub)problem? How would this be different? Cheers, Cesare > > > > > > > > On Tue, 6 Jun 2006, Leonardo de Moura wrote: > >> Hi Roberto, >> >> This is a good point. >> >>> there is an important question about SMT-COMP which apparently nobody >>> asked so far: do we explicitly require infinite precision from SMT >>> solvers on arithmetic-related theories? (I.e., RDL/IDL/LAR/LAI and >>> combinations.) >> >> I believe one needs to use multi precision arithmetic for solving some >> problems in >> LRA and LIA, but I agree with you that the benchmarks in RDL and IDL >> do not >> require infinite precision. >> >> I'm not sure if we should force/require people to use multi precision >> arithmetic. >> I think it is perfectly ok to use fixed precision, and return "unknown" >> when an overflow/underflow is detected. >> >> I also believe it is perfectly ok to enter an incomplete solver, and >> return >> "unknown" for formulas you know your solver is incomplete. >> For example, Aaron Stump did that last year. CVC is not complete >> for integers, and he returned "unknown" for potential "sat" instances. >> >> Anyway, I really interested in knowing what the rest of the SMT community >> thinks about this issue. >> >>> On the one hand, infinite precision prevents an SMT tool from >>> giving wrong answers on some problems. On the other hand, using >>> infinite precision significantly slows down the performances (around >>> a 3 factor in our experience). >> >> In my experiments, I noticed a 1.4 factor. I'm using GMP >> (www.swox.com/gmp/) >> for infinite precision. I guess you are using it too. :-) >> We found that memory management can be expensive with GMP, and a good >> trick >> is to create a pool of GMP numbers and reuse them for intermediate >> computations. >> >> Cheers, >> Leonardo. >> >> > From rseba at dit.unitn.it Tue Jun 13 07:40:00 2006 From: rseba at dit.unitn.it (Roberto Sebastiani) Date: Tue Jun 13 07:40:24 2006 Subject: [SMTCOMP] IMPORTANT: the infinite-precision issue In-Reply-To: <448DE5D5.7090805@cs.uiowa.edu> References: <448DE5D5.7090805@cs.uiowa.edu> Message-ID: Cesare, I'm not sure I got the point. On Tue, 13 Jun 2006, Cesare Tinelli wrote: > (...) >>> From a more principled perspective, I don't like the idea of customizing >>> tools for >> single instances, or of adopting tricks like >> "if infty run mathsat_infty else if 32 run mathsat_32 else run mathsat_64". >> > Fair enough. But I find it puzzling to hear this from a main MathSAT > developer. Isn't MathSAT's layered approach based exactly on the principle of > using the right (sub)solver for each specific input (sub)problem? Yes, but the decision is taken internally by looking at the (atoms in) the formula, without exploiting any extra information provided as input. > How would > this be different? Sorry if my statement was not clear: here I meant avoiding customizations based *on the input flag information* "32,64,infty". If somebody wants to decide whether to use 32, 64, or infty precision on generical formulas in a layered manner, or whatever other technique, I'm perfectly at ease with that. Roberto > > Cheers, > > > Cesare > > > >> >> >> >> >> >> >> >> On Tue, 6 Jun 2006, Leonardo de Moura wrote: >> >>> Hi Roberto, >>> >>> This is a good point. >>> >>>> there is an important question about SMT-COMP which apparently nobody >>>> asked so far: do we explicitly require infinite precision from SMT >>>> solvers on arithmetic-related theories? (I.e., RDL/IDL/LAR/LAI and >>>> combinations.) >>> >>> I believe one needs to use multi precision arithmetic for solving some >>> problems in >>> LRA and LIA, but I agree with you that the benchmarks in RDL and IDL do >>> not >>> require infinite precision. >>> >>> I'm not sure if we should force/require people to use multi precision >>> arithmetic. >>> I think it is perfectly ok to use fixed precision, and return "unknown" >>> when an overflow/underflow is detected. >>> >>> I also believe it is perfectly ok to enter an incomplete solver, and >>> return >>> "unknown" for formulas you know your solver is incomplete. >>> For example, Aaron Stump did that last year. CVC is not complete >>> for integers, and he returned "unknown" for potential "sat" instances. >>> >>> Anyway, I really interested in knowing what the rest of the SMT community >>> thinks about this issue. >>> >>>> On the one hand, infinite precision prevents an SMT tool from >>>> giving wrong answers on some problems. On the other hand, using infinite >>>> precision significantly slows down the performances (around a 3 factor in >>>> our experience). >>> >>> In my experiments, I noticed a 1.4 factor. I'm using GMP >>> (www.swox.com/gmp/) >>> for infinite precision. I guess you are using it too. :-) >>> We found that memory management can be expensive with GMP, and a good >>> trick >>> is to create a pool of GMP numbers and reuse them for intermediate >>> computations. >>> >>> Cheers, >>> Leonardo. >>> >>> >> > -- ------------------------------------------------------------------ Prof. ROBERTO SEBASTIANI Dip. Informatica e Telecomunicazioni Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba ------------------------------------------------------------------ From demoura at csl.sri.com Tue Jun 13 10:44:39 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Tue Jun 13 10:44:42 2006 Subject: [SMTCOMP] Benchmarks updated Message-ID: <6FDB7B68-2688-4F00-874C-9299702FB559@csl.sri.com> We updated the benchmarks at: http://www.csl.sri.com/users/demoura/ smt-comp/benchmarks.shtml The main modifications are: 1) removed the spurious ":extrasorts (ANY)" from the benchmarks in QF_LRA 2) included missing benchmarks from SMT-COMP'05: check directories and several directories in QF_IDL 3) removed benchmarks from wrong divisions: scheduling and skdmxa from QF_LRA 4) removed the spurious parenthesis from the smtlib-predabs benchmarks. From demoura at csl.sri.com Tue Jun 20 13:53:53 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Tue Jun 20 13:53:56 2006 Subject: [SMTCOMP] New benchmarks for QF_AUFLIA and AUFLIA Message-ID: <62C04722-C1EE-4003-A3B2-ECED36C5BBED@csl.sri.com> Hi, Aaron Bradley (Stanford) submitted several benchmarks for QF_AUFLIA and AUFLIA. The benchmarks were produced using his tool: piVC. We've also posted some benchmarks (AUFLIA) from the haRVey theorem prover. The new benchmarks can be downloaded at: http://www.csl.sri.com/users/demoura/smt-comp/benchmarks.shtml Leonardo From demoura at csl.sri.com Wed Jun 21 11:18:46 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Wed Jun 21 11:18:46 2006 Subject: [SMTCOMP] New benchmarks for QF_LIA Message-ID: Hi, Tobias Schuele submitted several benchmarks for QF_LIA. The benchmarks were generated using bounded model checking procedures. The Averest website is located at: http://www.averest.org. The new benchmarks can be downloaded at: http://www.csl.sri.com/users/demoura/smt-comp/benchmarks.shtml Leonardo From demoura at csl.sri.com Sat Jun 24 11:13:49 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Sat Jun 24 11:13:51 2006 Subject: [SMTCOMP] New QF_AUFLIA benchmarks Message-ID: Hi, We translated to SMT-LIB format a subset of the benchmarks used in the paper: "Big proof engines as little proof engines: new results on rewrite-based satisfiability procedure" Alessandro Armando, Maria Paola Bonacina, Silvio Ranise, Stephan Schulz. PDPAR'05 (http://www.ai.dist.unige.it/pdpar05/) The benchmarks can be downloaded at: http://www.csl.sri.com/users/demoura/smt-comp/benchmarks.shtml Leonardo From demoura at csl.sri.com Fri Jun 30 14:34:14 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Fri Jun 30 14:34:15 2006 Subject: [SMTCOMP] Revised benchmark library Message-ID: <49D3FE57-9728-4DCB-9856-581A6224717F@csl.sri.com> Hi, We posted the revised benchmark library at: http://www.csl.sri.com/users/demoura/smt-comp/benchmarks.shtml We are planning to post the benchmarks for QF_UFBV next week. We will *not* add more benchmarks to the old divisions. We will probably *not* add new benchmarks to AUFLIA and AUFLIRA. Our goal is to freeze the library next week. If you find problems with the benchmarks, please tell us and we will fix them for the final release on August 1st. The revised library contains the answer (:status attribute) for more than 99% of the benchmarks. If you know the answer for a benchmark marked as unknown, please tell us. The new library contains two new sets of benchmarks: NASA (AUFLIRA), and Simplify/ESC-Java (AUFLIA). We also included two new attributes: difficulty and category. These attributes will be used by the benchmark selection tool. Additional information and the benchmark selection tool are available at: http://www.csl.sri.com/users/demoura/smt-comp/bench_selection.shtml Best wishes, Leonardo for Aaron and Clark, too.