From oliveras at lsi.upc.edu Sun Jul 2 11:01:55 2006 From: oliveras at lsi.upc.edu (Albert Oliveras Llunell) Date: Sun Jul 2 09:02:13 2006 Subject: [SMTCOMP] The infinite-precision issue again In-Reply-To: References: Message-ID: <44A80A13.3060304@lsi.upc.edu> Dear colleagues, I followed with interest the discussion about the infinite-precision issue and was satisfied with the conclusions obtained. However, I am a little bit surprised after the final set of benchmarks has been published. My surprise concerns all divisions related with difference logic (QF_RDL, QF_IDL and QF_UFIDL). To the best of my knowledge most solvers for DL are based on negative-cycle detection or shortest-path computation. With all these algorithms, and unlike with Simplex-like methods, infinite precision is not an issue: if one adds all the integers appearing in the input formula and an overflow is not produced, then one can be sure that infinite-precision is not needed. And this is what happens in all benchmarks (correct me if I am wrong) in those divisions except for the artificially introduced benchmarks. As far as I know, the DL division was created because it had been widely proved that DL was powerful enough to express interesting problems but it was much more efficient than full linear arithmetic. Now, forcing infinite-precision in DL seems like incentivating people to develop less efficient solvers for a set of benchmarks (DL where infinite precision is needed) which, after two years of collecting benchmarks, is still empty. I was assuming when the final decision was made that this year there would be new real benchmarks with the infinite-precision requirement, but this does not seem to be the case. For all these reaons, I would suggest to remove the artificially added benchmarks from those divisions. However, of course, I am **very** interested in knowing what other people think about this issue. Best wishes, Albert From demoura at csl.sri.com Sun Jul 2 10:04:46 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Sun Jul 2 10:04:49 2006 Subject: [SMTCOMP] The infinite-precision issue again In-Reply-To: <44A80A13.3060304@lsi.upc.edu> References: <44A80A13.3060304@lsi.upc.edu> Message-ID: <0923852C-B3B2-4C71-98DB-BCFD1E823C7C@csl.sri.com> Hi Albert, > I followed with interest the discussion about the infinite- > precision issue and was satisfied with the conclusions obtained. > However, I am a little bit surprised after the final set of > benchmarks has been published. On June 10, I posted an email describing Aaron's, Clark's, and my position about this issue. In the end of the email, we said: >> That said, we will enforce that (where applicable) each division >> contains >> at least one benchmark that requires bignums, kind of like we did >> with >> integers last year. That way, people have some incentive to move to >> multi-precision arithmetic. Since nobody complained, we assumed everybody was happy with this decision. > For all these reaons, I would suggest to remove the artificially > added benchmarks from those divisions. However, of course, I am > **very** interested in knowing what other people think about this > issue. I understand your reasons, and I am also interested in knowing what others think about this issue. Cheers, Leonardo From Hyondeuk.Kim at colorado.edu Mon Jul 3 06:14:57 2006 From: Hyondeuk.Kim at colorado.edu (Hyondeuk.Kim@colorado.edu) Date: Mon Jul 3 06:15:02 2006 Subject: [SMTCOMP] The infinite-precision issue again In-Reply-To: <0923852C-B3B2-4C71-98DB-BCFD1E823C7C@csl.sri.com> References: <44A80A13.3060304@lsi.upc.edu> <0923852C-B3B2-4C71-98DB-BCFD1E823C7C@csl.sri.com> Message-ID: <1151932497.44a918514e899@webmail.colorado.edu> Hello Albert and Leonardo, I don't understand the difference between the original SMT benchmarks and the new submitted benchmarks. As Albert mentioned in his paper, the orignal benchmarks are from the real world and some of them are hand-maded. I believe that the new submitted benchmarks are also from the real world. I also believe that the current benchmark is somewhat biased; for example, most of them are unsat problem. SMT-LIB encourage people to submit new benchmarks. If the new set of benchmarks are not used in SMT COMP, this does not meet the objective of SMT LIB. I would like to hear your opinion about this. Thank you. Hyondeuk Quoting Leonardo de Moura : > Hi Albert, > > > > I followed with interest the discussion about the infinite- > > precision issue and was satisfied with the conclusions obtained. > > However, I am a little bit surprised after the final set of > > benchmarks has been published. > > On June 10, I posted an email describing Aaron's, Clark's, and my > position about this issue. > In the end of the email, we said: > > > >> That said, we will enforce that (where applicable) each division > >> contains > >> at least one benchmark that requires bignums, kind of like we did > >> with > >> integers last year. That way, people have some incentive to move to > >> multi-precision arithmetic. > > > Since nobody complained, we assumed everybody was happy with this > decision. > > > > For all these reaons, I would suggest to remove the artificially > > added benchmarks from those divisions. However, of course, I am > > **very** interested in knowing what other people think about this > > issue. > > I understand your reasons, and I am also interested in knowing what > others think about this > issue. > > Cheers, > Leonardo > > > > > > > From demoura at csl.sri.com Mon Jul 3 09:04:42 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Mon Jul 3 09:04:44 2006 Subject: [SMTCOMP] The infinite-precision issue again In-Reply-To: <1151932497.44a918514e899@webmail.colorado.edu> References: <44A80A13.3060304@lsi.upc.edu> <0923852C-B3B2-4C71-98DB-BCFD1E823C7C@csl.sri.com> <1151932497.44a918514e899@webmail.colorado.edu> Message-ID: Hi Hyondeuk, > I don't understand the difference between the original SMT > benchmarks and the > new submitted benchmarks. There is no difference between original and new benchmarks. The benchmark selection algorithm does not distinguish between old (original) and new benchmarks. > As Albert mentioned in his paper, the orignal > benchmarks are from the real world and some of them are hand-maded. > I believe > that the new submitted benchmarks are also from the real world. Some of the new benchmarks are also from the real-world and some of them are also hand-maded. > I also believe > that the current benchmark is somewhat biased; for example, most of > them are > unsat problem. I agree SMT-LIB has more unsat than sat benchmarks, but several new SAT instances were included this year. Moreover, the benchmark selection algorithm tries to produce (when possible) a balanced distribution of benchmarks: 25% easy-sat, 25% easy-unsat, 25% hard-sat, 25% hard-unsat. > SMT-LIB encourage people to submit new benchmarks. If the new > set of benchmarks are not used in SMT COMP, this does not meet the > objective > of SMT LIB. We are using all submitted benchmarks. If some benchmarks are missing from the revised library, then please tell us. Leonardo From demoura at csl.sri.com Wed Jul 5 14:38:41 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Wed Jul 5 14:38:43 2006 Subject: [SMTCOMP] Revised benchmark library: updated Message-ID: Hi, We updated the benchmark library with the following modifications: 1) removed the spurious ":extrasorts (ANY)" from the benchmarks in QF_RDL and QF_IDL 2) fixed invalid benchmark id in QF_IDL/misc/2ba.smt 3) fixed a problem in QF_IDL reported by Hyondeuk Kim. Some benchmarks contained bound constraints (e.g., "(= x_229 1)"). This kind of constraint is not allowed in the difference logic divisions. Leonardo From demoura at csl.sri.com Wed Jul 5 14:54:28 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Wed Jul 5 14:54:30 2006 Subject: [SMTCOMP] scrambler updated Message-ID: Hi, The scrambler was updated. It was not recognizing Array1 and Array2 as builtin sorts in AUFLIRA. Some solvers were not distinguishing identifiers such as: $x20, ?x20, and x20. Thus, the new scrambler uses different prefixes for these identifiers: "$p", "?t", and "x". Leonardo From stump at cse.wustl.edu Thu Jul 6 09:06:35 2006 From: stump at cse.wustl.edu (Aaron Stump) Date: Thu Jul 6 09:06:40 2006 Subject: [SMTCOMP] The infinite-precision issue again In-Reply-To: <44A80A13.3060304@lsi.upc.edu> References: <44A80A13.3060304@lsi.upc.edu> Message-ID: <32216.1152201995@cse.wustl.edu> Hi, Albert. Clark, Leonardo, and I have been discussing the concerns you raised about having benchmarks crafted to test soundness when working with large numerals (I suggest calling these "artificial bignum benchmarks"). As Leonardo mentioned, we concluded the original discussion (sparked by Roberto Sebastiani) by saying that we would indeed include artifical bignum benchmarks in all arithmetic divisions. So we would really need to stay with that decision at this point, since we had no complaints and the competition is coming up soon. But addressing the issues you raised more directly, I would offer the following argument for why there should be no issue in including artificial bignum benchmarks. Solvers using machine arithmetic only (let's call these "32-bit solvers"), without a bignum package, are presumably doing so because their implementors believe this will confer a speed advantage. If that is correct, there should be benchmarks that can be solved within whatever time limit by 32-bit solvers, but not by similar solvers using a bignum package ("bignum solvers"). In that case, any points a 32-bit solver loses due to inability to perform arithmetic on large numerals should be recovered on benchmarks where the cost of arithmetic on numerals is decisive for solving or not solving the benchmark. Now the reality is, no one seems to have any such benchmarks, where the cost of arithmetic on numerals is the deciding factor. Certainly no one has provided them to SMT-COMP. So for competition, bignum solvers may indeed have an advantage. The burden is, in my opinion, on the proponent of 32-bit solvers to provide benchmarks where using native arithmetic is essential to solve the benchmark in time. Of course, if it is true that 32-bit arithmetic confers a significant advantage, more robust solutions are to use 32-bit arithmetic and fail over to bignum arithmetic if overflow/underflow is detected; or possibly conservatively analyze the benchmark to decide which arithmetic package to use (though this may not be trivial to determine). Finally, even if the jury is still out on whether native arithmetic confers a crucial advantage on benchmarks of interest, supporting arbitrary precision numerals is more robust, and hence requiring such support seems like a reasonable default position in the absence of more evidence. Aaron, attempting to summarize the position also of Clark and Leonardo From barrett at cs.nyu.edu Thu Jul 6 10:18:50 2006 From: barrett at cs.nyu.edu (Clark Barrett) Date: Thu Jul 6 10:13:10 2006 Subject: [SMTCOMP] The infinite-precision issue again In-Reply-To: <32216.1152201995@cse.wustl.edu> Message-ID: <200607061718.k66HIoKl032360@zaphod.cs.nyu.edu> I just wanted to add a point that Leonardo brought up: > (paraphrasing...) > Anyone entering QF_LIA and QF_LRA will need bignum support because there are > actual benchmarks that require it. Thus it should be possible to > to use the linear arithmetic procedure when the check for fixnums fails in a > difference logic benchmark. Because this seems like an easy fix, I am inclined to keep the current scheme as it encourages everyone to migrate towards a more robust solution. -Clark From rseba at dit.unitn.it Fri Jul 7 10:00:03 2006 From: rseba at dit.unitn.it (Roberto Sebastiani) Date: Fri Jul 7 10:00:20 2006 Subject: [SMTCOMP] About the QF_IDL "n-queens" problms Message-ID: Dear colleagues, Let me raise one more point, about "n-queens" problems in the QF_IDL benchmark suite. (I apologize for doing it only now, but we noticed this fact only recently.) 297 out of 893 total benchmarks in QF_IDL are "n-queens". It seems to us they are all "pigeonhole-style" problems: 1. { x_i != x_j }_{ij} 2. { x_i=value_{i_1} | x_i=value_{i_2} | ... | x_i=value_{i_K} }_i which are encoded into QF_IDL as 1.1. { x_i != x_j }_{ij} 2.1. { {i_1} <= x_i-zero <= i_K }_i ("zero" being one variable representing zero). Thus, they are FULLY-COMBINATORIAL CSP PROBLEMS disguised as SMT(QF_IDL) problems, and have hardly anything to do with QF_IDL. In fact, no arithmetical property should be exploited to solve these problems. (Notice, e.g., that the actual values of the constants value_{i_j}'s are irrelevant: the only relevant fact is weather they are pairwise equal or different. Thus. "-" and ">=" have no effective mathematical meaning here.) We believe this is against the spirit of SMT-COMP. Moreover, these are the kind of problems which may push people to customize "ad hoc" solutions for the competition. Thus, we believe these problems should be dropped from the list. Roberto -- ------------------------------------------------------------------ Prof. ROBERTO SEBASTIANI Dip. Informatica e Telecomunicazioni Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba ------------------------------------------------------------------ From Hyondeuk.Kim at colorado.edu Fri Jul 7 11:10:08 2006 From: Hyondeuk.Kim at colorado.edu (Hyondeuk.Kim@colorado.edu) Date: Fri Jul 7 11:10:15 2006 Subject: [SMTCOMP] About the QF_IDL "n-queens" problms In-Reply-To: References: Message-ID: <1152295808.44aea380256eb@webmail.colorado.edu> Hello Roberto, I submitted this benchmark since it is interesting to know how difference logic solver handles the problem with many disequalities. I think each set of SMT benchmarks in QF_IDL has a different feature. For example, diamond suite makes the solver to check every diamond for feasibility, mathsat suite hardly makes the solver to call theory solver. As I mentioned, the purpose of the queens benchmark is to check how the solver handles the disequality constraints. Therefore, I think it is still interesting to leave some queens benchmarks in the SMT competition. Thank you. Hyondeuk Quoting Roberto Sebastiani : > Dear colleagues, > > Let me raise one more point, about "n-queens" problems in > the QF_IDL benchmark suite. > (I apologize for doing it only now, but we noticed this fact > only recently.) > > 297 out of 893 total benchmarks in QF_IDL are "n-queens". > It seems to us they are all "pigeonhole-style" problems: > > 1. { x_i != x_j }_{ij} > 2. { x_i=value_{i_1} | x_i=value_{i_2} | ... | x_i=value_{i_K} }_i > > which are encoded into QF_IDL as > > 1.1. { x_i != x_j }_{ij} > 2.1. { {i_1} <= x_i-zero <= i_K }_i > > ("zero" being one variable representing zero). > > Thus, they are FULLY-COMBINATORIAL CSP PROBLEMS disguised as > SMT(QF_IDL) problems, and have hardly anything to do with QF_IDL. > In fact, no arithmetical property should be exploited to solve > these problems. > (Notice, e.g., that the actual values of the constants > value_{i_j}'s are irrelevant: the only relevant fact is weather > they are pairwise equal or different. Thus. "-" and ">=" have no > effective mathematical meaning here.) > > We believe this is against the spirit of SMT-COMP. > Moreover, these are the kind of problems which may push people > to customize "ad hoc" solutions for the competition. > > Thus, we believe these problems should be dropped from the list. > > Roberto > > > -- > ------------------------------------------------------------------ > Prof. ROBERTO SEBASTIANI > Dip. Informatica e Telecomunicazioni > Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 > Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 > roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba > ------------------------------------------------------------------ > From Hyondeuk.Kim at colorado.edu Fri Jul 7 11:45:54 2006 From: Hyondeuk.Kim at colorado.edu (Hyondeuk.Kim@colorado.edu) Date: Fri Jul 7 11:45:58 2006 Subject: [SMTCOMP] About the QF_IDL "n-queens" problms In-Reply-To: References: Message-ID: <1152297954.44aeabe2c0984@webmail.colorado.edu> Hello, I think I did not reply to all. This is what I replied to Roberto. I submitted Queens benchmark since it is interesting to know how difference logic solver handles the problem with many disequality constraints. As you know, in QF_IDL, each benchmark set has its own feature. For example, the diamond benchmark makes the solver to check every diamonds for feasibility, and the mathsat benchmark makes the solver hardly call the theory solver. As I said, the purpose of the queens benchmark is to check how the solver handles the diseqaulity constraints. Therefore, I think it is still interesting to leave Queens benchmarks in SMT COMP. Thank you. Hyondeuk Quoting Roberto Sebastiani : > Dear colleagues, > > Let me raise one more point, about "n-queens" problems in > the QF_IDL benchmark suite. > (I apologize for doing it only now, but we noticed this fact > only recently.) > > 297 out of 893 total benchmarks in QF_IDL are "n-queens". > It seems to us they are all "pigeonhole-style" problems: > > 1. { x_i != x_j }_{ij} > 2. { x_i=value_{i_1} | x_i=value_{i_2} | ... | x_i=value_{i_K} }_i > > which are encoded into QF_IDL as > > 1.1. { x_i != x_j }_{ij} > 2.1. { {i_1} <= x_i-zero <= i_K }_i > > ("zero" being one variable representing zero). > > Thus, they are FULLY-COMBINATORIAL CSP PROBLEMS disguised as > SMT(QF_IDL) problems, and have hardly anything to do with QF_IDL. > In fact, no arithmetical property should be exploited to solve > these problems. > (Notice, e.g., that the actual values of the constants > value_{i_j}'s are irrelevant: the only relevant fact is weather > they are pairwise equal or different. Thus. "-" and ">=" have no > effective mathematical meaning here.) > > We believe this is against the spirit of SMT-COMP. > Moreover, these are the kind of problems which may push people > to customize "ad hoc" solutions for the competition. > > Thus, we believe these problems should be dropped from the list. > > Roberto > > > -- > ------------------------------------------------------------------ > Prof. ROBERTO SEBASTIANI > Dip. Informatica e Telecomunicazioni > Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 > Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 > roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba > ------------------------------------------------------------------ > From demoura at csl.sri.com Fri Jul 7 12:10:36 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Fri Jul 7 12:10:36 2006 Subject: [SMTCOMP] About the QF_IDL "n-queens" problms In-Reply-To: References: Message-ID: <12FB4F26-615C-4797-B4EC-6C60BAE9E198@csl.sri.com> Dear Roberto, We are aware of the problem. The n-queens benchmarks were marked as crafted. The benchmark selection algorithm uses the following distribution: 85% industrial, 10% crafted, 5% random. In our simulations, only 4 queens problems were selected. You can find more information about the benchmark selection algorithm at: http://www.csl.sri.com/users/demoura/smt-comp/bench_selection.shtml Cheers, Leonardo On Jul 7, 2006, at 10:00 AM, Roberto Sebastiani wrote: > Dear colleagues, > > Let me raise one more point, about "n-queens" problems in the > QF_IDL benchmark suite. (I apologize for doing it only now, but we > noticed this fact > only recently.) > > 297 out of 893 total benchmarks in QF_IDL are "n-queens". > It seems to us they are all "pigeonhole-style" problems: > > 1. { x_i != x_j }_{ij} > 2. { x_i=value_{i_1} | x_i=value_{i_2} | ... | x_i=value_{i_K} }_i > > which are encoded into QF_IDL as > > 1.1. { x_i != x_j }_{ij} > 2.1. { {i_1} <= x_i-zero <= i_K }_i > > ("zero" being one variable representing zero). > > Thus, they are FULLY-COMBINATORIAL CSP PROBLEMS disguised as SMT > (QF_IDL) problems, and have hardly anything to do with QF_IDL. > In fact, no arithmetical property should be exploited to solve > these problems. > (Notice, e.g., that the actual values of the constants value_ > {i_j}'s are irrelevant: the only relevant fact is weather they are > pairwise equal or different. Thus. "-" and ">=" have no effective > mathematical meaning here.) > > We believe this is against the spirit of SMT-COMP. > Moreover, these are the kind of problems which may push people to > customize "ad hoc" solutions for the competition. > > Thus, we believe these problems should be dropped from the list. > > Roberto > > > -- > ------------------------------------------------------------------ > Prof. ROBERTO SEBASTIANI > Dip. Informatica e Telecomunicazioni > Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 > Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 > roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba > ------------------------------------------------------------------ From demoura at csl.sri.com Fri Jul 7 19:23:08 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Fri Jul 7 19:23:12 2006 Subject: [SMTCOMP] Benchmark library was updated Message-ID: <43E14D5F-0ADC-480F-8A61-652F11CD677F@csl.sri.com> Hi, We updated the benchmark library. http://www.csl.sri.com/users/demoura/smt-comp/benchmarks.shtml The main modifications are: - the bitvector (QF_UFBV32) benchmarks are available - most of the Averest benchmarks in QF_LIA were in the difference logic fragment. So, we move them to QF_IDL. We are officially freezing the benchmark library. No new benchmark will be included in the library. Best wishes, Leonardo for Aaron and Clark, too. From rseba at dit.unitn.it Mon Jul 10 02:14:01 2006 From: rseba at dit.unitn.it (Roberto Sebastiani) Date: Mon Jul 10 02:14:14 2006 Subject: [SMTCOMP] About the QF_IDL "n-queens" problms In-Reply-To: <1152295808.44aea380256eb@webmail.colorado.edu> References: <1152295808.44aea380256eb@webmail.colorado.edu> Message-ID: Hi Hyondeuk I'm at ease with SMT(QF_IDL) problems with many disequalities. The problem is that n-queens are *CSP* problems with many disequalities! As I said below, the risk with this kind of problems is that, (unlike diamonds & mathsat/fischer suites) these problems have a recognizable structure, so that ad hoc solutions can be adopted for the competition. Roberto PS: As far as the diamonds & mathsat/fischer suites are concerned, I'm open to discuss the opportunity of dropping them as well if anybody proposes it. On Fri, 7 Jul 2006, Hyondeuk.Kim@colorado.edu wrote: > Hello Roberto, > > I submitted this benchmark since it is interesting to know how difference logic > solver handles the problem with many disequalities. > I think each set of SMT > benchmarks in QF_IDL has a different feature. For example, diamond suite makes > the solver to check every diamond for feasibility, mathsat suite hardly makes > the solver to call theory solver. As I mentioned, the purpose of the queens > benchmark is to check how the solver handles the disequality constraints. > Therefore, I think it is still interesting to leave some queens benchmarks in > the SMT competition. > > Thank you. > > Hyondeuk > > Quoting Roberto Sebastiani : > >> Dear colleagues, >> >> Let me raise one more point, about "n-queens" problems in >> the QF_IDL benchmark suite. >> (I apologize for doing it only now, but we noticed this fact >> only recently.) >> >> 297 out of 893 total benchmarks in QF_IDL are "n-queens". >> It seems to us they are all "pigeonhole-style" problems: >> >> 1. { x_i != x_j }_{ij} >> 2. { x_i=value_{i_1} | x_i=value_{i_2} | ... | x_i=value_{i_K} }_i >> >> which are encoded into QF_IDL as >> >> 1.1. { x_i != x_j }_{ij} >> 2.1. { {i_1} <= x_i-zero <= i_K }_i >> >> ("zero" being one variable representing zero). >> >> Thus, they are FULLY-COMBINATORIAL CSP PROBLEMS disguised as >> SMT(QF_IDL) problems, and have hardly anything to do with QF_IDL. >> In fact, no arithmetical property should be exploited to solve >> these problems. >> (Notice, e.g., that the actual values of the constants >> value_{i_j}'s are irrelevant: the only relevant fact is weather >> they are pairwise equal or different. Thus. "-" and ">=" have no >> effective mathematical meaning here.) >> >> We believe this is against the spirit of SMT-COMP. >> Moreover, these are the kind of problems which may push people >> to customize "ad hoc" solutions for the competition. >> >> Thus, we believe these problems should be dropped from the list. >> >> Roberto >> >> >> -- >> ------------------------------------------------------------------ >> Prof. ROBERTO SEBASTIANI >> Dip. Informatica e Telecomunicazioni >> Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 >> Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 >> roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba >> ------------------------------------------------------------------ >> > > -- ------------------------------------------------------------------ Prof. ROBERTO SEBASTIANI Dip. Informatica e Telecomunicazioni Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba ------------------------------------------------------------------ From rseba at dit.unitn.it Mon Jul 10 02:34:40 2006 From: rseba at dit.unitn.it (Roberto Sebastiani) Date: Mon Jul 10 02:34:45 2006 Subject: [SMTCOMP] About the QF_IDL "n-queens" problms In-Reply-To: <12FB4F26-615C-4797-B4EC-6C60BAE9E198@csl.sri.com> References: <12FB4F26-615C-4797-B4EC-6C60BAE9E198@csl.sri.com> Message-ID: Hi Leonardo, see my answer to Hyondeuk. MOreover: On Fri, 7 Jul 2006, Leonardo de Moura wrote: > Dear Roberto, > > We are aware of the problem. The n-queens benchmarks were marked > as crafted. The benchmark selection algorithm uses the following > distribution: 85% industrial, 10% crafted, 5% random. > In our simulations, only 4 queens problems were selected. Well, 4 problems are well enough to reshape the whole result, in particular on QF_IDL. E.g., last year: TOOL #SOLVED BarcelogicTools 47 Yices 47 MathSat 46 Ario 43 Roberto > You can find more information about the benchmark selection > algorithm at: > http://www.csl.sri.com/users/demoura/smt-comp/bench_selection.shtml > > Cheers, > Leonardo > > > On Jul 7, 2006, at 10:00 AM, Roberto Sebastiani wrote: > >> Dear colleagues, >> >> Let me raise one more point, about "n-queens" problems in the QF_IDL >> benchmark suite. (I apologize for doing it only now, but we noticed this >> fact >> only recently.) >> >> 297 out of 893 total benchmarks in QF_IDL are "n-queens". >> It seems to us they are all "pigeonhole-style" problems: >> >> 1. { x_i != x_j }_{ij} >> 2. { x_i=value_{i_1} | x_i=value_{i_2} | ... | x_i=value_{i_K} }_i >> >> which are encoded into QF_IDL as >> >> 1.1. { x_i != x_j }_{ij} >> 2.1. { {i_1} <= x_i-zero <= i_K }_i >> >> ("zero" being one variable representing zero). >> >> Thus, they are FULLY-COMBINATORIAL CSP PROBLEMS disguised as SMT(QF_IDL) >> problems, and have hardly anything to do with QF_IDL. >> In fact, no arithmetical property should be exploited to solve these >> problems. >> (Notice, e.g., that the actual values of the constants value_{i_j}'s are >> irrelevant: the only relevant fact is weather they are pairwise equal or >> different. Thus. "-" and ">=" have no effective mathematical meaning here.) >> >> We believe this is against the spirit of SMT-COMP. >> Moreover, these are the kind of problems which may push people to customize >> "ad hoc" solutions for the competition. >> >> Thus, we believe these problems should be dropped from the list. >> >> Roberto >> >> >> -- >> ------------------------------------------------------------------ >> Prof. ROBERTO SEBASTIANI >> Dip. Informatica e Telecomunicazioni >> Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 >> Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 >> roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba >> ------------------------------------------------------------------ -- ------------------------------------------------------------------ Prof. ROBERTO SEBASTIANI Dip. Informatica e Telecomunicazioni Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba ------------------------------------------------------------------ From oliveras at lsi.upc.edu Mon Jul 10 05:23:24 2006 From: oliveras at lsi.upc.edu (Albert Oliveras Llunell) Date: Mon Jul 10 05:22:57 2006 Subject: [SMTCOMP] The infinite-precision issue again In-Reply-To: <32216.1152201995@cse.wustl.edu> References: <44A80A13.3060304@lsi.upc.edu> <32216.1152201995@cse.wustl.edu> Message-ID: <44B246BC.5080906@lsi.upc.edu> Hi all, first of all I would like to thank you for your detailed answer. These are my reactions: >So we would really need to stay with that decision at this point, since >we had no complaints and the competition is coming up soon. > > Sure, I completely understand and accept that. >Finally, even if the jury is still out on whether native arithmetic >confers a crucial advantage on benchmarks of interest, supporting >arbitrary precision numerals is more robust, and hence requiring such >support seems like a reasonable default position in the absence of >more evidence. > > > Of course, if I had to choose between robustness and efficiency, I would definetely choose robustness. However, the point I wanted to make is that we should be careful about losing efficiency by incorporating extensions that do not lead to a real gain in robustness. I thought that big number support could be problematic in this direction, and this is why I raised this point. However, I am not religious about that. I fully understand your explanations and indeed they seem quite reasonable to me. I hope to meet you all at Seattle and discuss about this and other issues. Cheers Albert -- _______________________________________________________________________ Albert Oliveras Llunell www.lsi.upc.edu/~oliveras Technical University of Catalonia (UPC) oliveras@lsi.upc.edu Dpt. Llenguatges i Sistemes Informatics(LSI) Phone: +34-93-4137861 Jordi Girona 1, E-08034 Barcelona, Spain Fax: +34-93-4137033 _______________________________________________________________________ From demoura at csl.sri.com Mon Jul 10 06:42:41 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Mon Jul 10 06:42:44 2006 Subject: [SMTCOMP] About the QF_IDL "n-queens" problms In-Reply-To: References: <12FB4F26-615C-4797-B4EC-6C60BAE9E198@csl.sri.com> Message-ID: <1B9638B3-6BA7-4199-90CF-9AF7B3AB6003@csl.sri.com> Hi Roberto, > Well, 4 problems are well enough to reshape the whole result, > in particular on QF_IDL. E.g., last year: This is a good point. BTW, in my previous email, I forgot to mention we are considering 100 benchmarks per division in the our simulations. So, it is 4 queens benchmarks out of 100. Cheers, Leonardo. From rseba at dit.unitn.it Mon Jul 10 08:08:49 2006 From: rseba at dit.unitn.it (Roberto Sebastiani) Date: Mon Jul 10 08:08:53 2006 Subject: [SMTCOMP] About the QF_IDL "n-queens" problms In-Reply-To: <1B9638B3-6BA7-4199-90CF-9AF7B3AB6003@csl.sri.com> References: <12FB4F26-615C-4797-B4EC-6C60BAE9E198@csl.sri.com> <1B9638B3-6BA7-4199-90CF-9AF7B3AB6003@csl.sri.com> Message-ID: On Mon, 10 Jul 2006, Leonardo de Moura wrote: > Hi Roberto, > >> Well, 4 problems are well enough to reshape the whole result, >> in particular on QF_IDL. E.g., last year: > > This is a good point. > > BTW, in my previous email, I forgot to mention we are considering > 100 benchmarks per division in the our simulations. So, it is > 4 queens benchmarks out of 100. if 10% of randomly selected problems are crafted, and most of them are n-queens, we may have up of 10 such problems. Roberto PS: when you say that 85% of problems are industrial, 10% crafted and 5% random, you mean: a) that exactly 85%, 10%, 5% of the picked problems will be industrial, crafted, random respectively b) that each problem will be picked among industrial, crafted and random with probability .85, .1, and .05 respectively I understand option a), right? > > Cheers, > Leonardo. > > > > -- ------------------------------------------------------------------ Prof. ROBERTO SEBASTIANI Dip. Informatica e Telecomunicazioni Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba ------------------------------------------------------------------ From demoura at csl.sri.com Mon Jul 10 09:48:18 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Mon Jul 10 09:48:21 2006 Subject: [SMTCOMP] About the QF_IDL "n-queens" problms In-Reply-To: References: <12FB4F26-615C-4797-B4EC-6C60BAE9E198@csl.sri.com> <1B9638B3-6BA7-4199-90CF-9AF7B3AB6003@csl.sri.com> Message-ID: <13D501EA-9F9A-48C2-AA68-36B625108D36@csl.sri.com> Hi Roberto, > if 10% of randomly selected problems are crafted, and most of them > are n-queens, we may have up of 10 such problems. There are other crafted problems (e.g., diamonds and jobshop) in QF_IDL. So, I don't think we will end up with 10 queens benchmarks, but 5 or 6 is a real possibility. > PS: when you say that 85% of problems are industrial, 10% crafted > and 5% random, you mean: > a) that exactly 85%, 10%, 5% of the picked problems will be > industrial, > crafted, random respectively Yes, I meant option a) Leonardo From barrett at cs.nyu.edu Tue Jul 11 08:54:48 2006 From: barrett at cs.nyu.edu (Clark Barrett) Date: Tue Jul 11 08:48:50 2006 Subject: [SMTCOMP] Re: QF_UFBV: on last-minute extension with sign In-Reply-To: <20060711100015.GA3282@itc.it> Message-ID: <200607111554.k6BFsmkx005011@zaphod.cs.nyu.edu> > However, I think that the extension of QF_UFBV32 with signed operators > should not be accepted at this stage of the competition. > > In fact, the bit vectors theory has been fixed a long time ago --- the > description was posted on the list by Cesare on May 5th. The > description made it clear that the interpretation would be > unsigned. That signed arithmetic would not be part of the theory had > been discussed and agreed upon with Cesare, Silvio, and others. > > No variations were communicated on the topic since. Thus, the > competitors entering this category had no reason to include signed > operators. A last minute addition to the theory would give them a > disadvantage against those who have signed arithmetic operators > already available, or somehow knew that the extension was being taken > into consideration. > > Notice that this is really a last minute extension: for the other > categories, not only the theories, but also the benchmarks, have been > frozen more than a week ago. Let me comment on why this happened. When we first formalized the bit-vector theory, we anticipated receiving benchmarks from industry that would fit nicely in the defined theory and logic. Unfortunately, these benchmarks never came. When it became clear they would not be available, I went looking for other benchmarks. Stanford offered the ones that I ultimately translated. Unfortunately, most of them required the use of signed arithmetic as well as arrays. One important aim of SMT-LIB is to capture benchmarks in their native form and avoid as much as possible translations and re-interpretations, so I was reluctant to make any changes to the theories and signatures used by the benchmarks. In fact, I was contemplating adding a theory combining bitvectors with arrays to properly translate the benchmarks, but upon closer examination, I realized that in most cases, arrays were simply being used like functions (i.e. the update/store/write feature was not being used). Thus, I felt that it was more accurate to represent these as uninterpreted functions and I made that translation. A similar translation could have been done to eliminate the signed arithmetic operations, but I felt that would be against the spirit of SMT-LIB because these are common bit-vector operations, and they should be included in the logic instead. I am happy with the benchmarks as they are and feel that they are an accurate representation of the original application and a good addition to SMT-LIB. The question of whether they should be included in SMT-COMP is somewhat orthogonal, but related. I tend to think they should be included for the following reasons: 1) First, let me clear up a misunderstanding resulting from a typo in the previous version of the logic (see my previous email). It is actually the case that 6819 of 8247 files (83%) use the signed operators, not just 437. Perhaps there is still enough variation in the remaining benchmarks, but I don't know. By the way, I suspect this also explains Alessandro's note about undeclared symbols. 2) The signed comparison and extend operators can be expressed fairly simply in terms of the previously existing operators, so it shouldn't be difficult to add the extensions. 3) One possible criticism is that "native" support for signed operators will be more efficient than implementing them as existing operators, but for my own part, I can say that I expect my system would have the same performance and I don't know of any system that has special performance capabilities for signed operations. That said, I am interested in what others think. Should we use the full set of benchmarks for SMT-COMP, or restrict it to the 17% of them that do not use the signed operators? I suggest that the remainder of this discussion take place on the smt-comp list rather than the smt-lib list. Clark Barrett From vganesh at stanford.edu Tue Jul 11 14:54:32 2006 From: vganesh at stanford.edu (Vijay Ganesh) Date: Wed Jul 12 08:18:46 2006 Subject: [SMTCOMP] Re: [SMT-LIB] QF_UFBV: on last-minute extension with sign In-Reply-To: <200607111554.k6BFsmkx005011@zaphod.cs.nyu.edu> References: <200607111554.k6BFsmkx005011@zaphod.cs.nyu.edu> Message-ID: Hi all, I agree with Clark's statement that signed operators can be easily translated into pre-existing operators. Furthermore, I think that signed operators do not provide any new oppurtunity for optimizations. If any such optimizations present themselves, then corresponding unsigned optimizations exist. Thanks & Warm Regards, Vijay Ganesh. From demoura at csl.sri.com Wed Jul 12 10:35:46 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Wed Jul 12 10:35:49 2006 Subject: [SMTCOMP] scrambler updated Message-ID: Hi, The scrambler was updated. The new version supports the QF_UFBV32 benchmarks. http://www.csl.sri.com/users/demoura/smt-comp/scrambler.shtml Leonardo From demoura at csl.sri.com Tue Jul 18 14:18:23 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Tue Jul 18 14:18:23 2006 Subject: [SMTCOMP] SMT-COMP info Message-ID: <14271F80-E022-44F7-B09E-8A76AED39342@csl.sri.com> Hi, Some additional information regarding SMT-COM'06: - We are planning to use 100 benchmarks/division and a 20 minutes timeout. - If you didn't pre-register yet, please pre-register as soon as possible. - Execution environment: OS: GNU/Linux version 2.6.14 Processor: Pentium 4, 3.00 GHz Memory: 2 Gb (1.5 GB memory limit for solver processes enforced) Cache: 2 Mb GCC: version 4.0.2 - The scrambler was updated again. Best wishes, Leonardo for Aaron and Clark too. From alberto.griggio at dit.unitn.it Fri Jul 21 06:59:43 2006 From: alberto.griggio at dit.unitn.it (Alberto Griggio) Date: Fri Jul 21 06:57:38 2006 Subject: [SMTCOMP] a question on the benchmark scrambler Message-ID: <200607211559.43967.alberto.griggio@dit.unitn.it> Hello all, I just downloaded and had a look at the benchmark scrambler for the competiton. I'm not a scheme expert, but from what I understand both from the source code and the output it generates, it seems that the scrambler doesn't permute equalities. Am I right or not? And if I'm right, is there any particular reason why this is not done? Not that I'm saying that this must (or should) be done, it is more a curiosity... Thanks, and regards, Alberto From demoura at csl.sri.com Fri Jul 21 07:47:03 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Fri Jul 21 07:47:04 2006 Subject: [SMTCOMP] a question on the benchmark scrambler In-Reply-To: <200607211559.43967.alberto.griggio@dit.unitn.it> References: <200607211559.43967.alberto.griggio@dit.unitn.it> Message-ID: Hi Alberto, > Am I right or not? Yes you are correct. > And if I'm right, is there any particular reason why this is not > done? Not that > I'm saying that this must (or should) be done, it is more a > curiosity... I didn't do that because in QF_RDL and QF_IDL the linear arithmetic atoms should have a specific format: - (op (- x y) c), - (op (- x y) (~ c)), where op is "=, <, <=, >=, >" I didn't want to change this format in the output produced by the scrambler because some tools may assume it. Anyway, if nobody depends on this restriction I can modify the scrambler to permute equalities. Leonardo From rseba at dit.unitn.it Fri Jul 21 07:55:58 2006 From: rseba at dit.unitn.it (Roberto Sebastiani) Date: Fri Jul 21 07:56:05 2006 Subject: [SMTCOMP] a question on the benchmark scrambler In-Reply-To: References: <200607211559.43967.alberto.griggio@dit.unitn.it> Message-ID: On Fri, 21 Jul 2006, Leonardo de Moura wrote: > Hi Alberto, > >> Am I right or not? > > Yes you are correct. > >> And if I'm right, is there any particular reason why this is not done? Not >> that >> I'm saying that this must (or should) be done, it is more a >> curiosity... > > I didn't do that because in QF_RDL and QF_IDL the linear arithmetic > atoms should have a specific format: > > - (op (- x y) c), > - (op (- x y) (~ c)), > > where op is "=, <, <=, >=, >" > > I didn't want to change this format in the output produced by the scrambler > because some tools may assume it. > > Anyway, if nobody depends on this restriction I can modify the scrambler > to permute equalities. I suspect barcelogic does depends on this restrictio. Am I wrong? > > Leonardo -- ------------------------------------------------------------------ Prof. ROBERTO SEBASTIANI Dip. Informatica e Telecomunicazioni Fac. Scienze M.F.N., Universita` di Trento Tel: +39 0461 881514 Via Sommarive 14, I-38050, Trento, Italy Fax: +39 0461 882093 roberto.sebastiani@dit.unitn.it http://www.dit.unitn.it/~rseba ------------------------------------------------------------------ From Hyondeuk.Kim at colorado.edu Fri Jul 21 08:23:49 2006 From: Hyondeuk.Kim at colorado.edu (Hyondeuk.Kim@colorado.edu) Date: Fri Jul 21 08:23:59 2006 Subject: [SMTCOMP] a question on the benchmark scrambler In-Reply-To: References: <200607211559.43967.alberto.griggio@dit.unitn.it> Message-ID: <1153495429.44c0f185d3222@webmail.colorado.edu> Hi Leonardo, I wonder how equalities can be permuted. Thank you. Hyondeuk Quoting Leonardo de Moura : > Hi Alberto, > > > Am I right or not? > > Yes you are correct. > > > And if I'm right, is there any particular reason why this is not > > done? Not that > > I'm saying that this must (or should) be done, it is more a > > curiosity... > > I didn't do that because in QF_RDL and QF_IDL the linear arithmetic > atoms should have a specific format: > > - (op (- x y) c), > - (op (- x y) (~ c)), > > where op is "=, <, <=, >=, >" > > I didn't want to change this format in the output produced by the > scrambler > because some tools may assume it. > > Anyway, if nobody depends on this restriction I can modify the scrambler > to permute equalities. > > Leonardo > From demoura at csl.sri.com Fri Jul 21 08:28:15 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Fri Jul 21 08:28:16 2006 Subject: [SMTCOMP] a question on the benchmark scrambler In-Reply-To: <1153495429.44c0f185d3222@webmail.colorado.edu> References: <200607211559.43967.alberto.griggio@dit.unitn.it> <1153495429.44c0f185d3222@webmail.colorado.edu> Message-ID: Hi Hyondeuk, > I wonder how equalities can be permuted. I meant the following transformation: (= (- x y) c) ---> (= c (- x y)) Leonardo From demoura at csl.sri.com Fri Jul 21 08:45:05 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Fri Jul 21 08:45:06 2006 Subject: [SMTCOMP] a question on the benchmark scrambler In-Reply-To: References: <200607211559.43967.alberto.griggio@dit.unitn.it> Message-ID: Hi Roberto, > I suspect barcelogic does depends on this restrictio. Am I wrong? I don't know which tools depend on this restriction, but I don't see a problem with them since the SMT-LIB specifies this restriction. Leonardo From oliveras at lsi.upc.edu Fri Jul 21 09:42:15 2006 From: oliveras at lsi.upc.edu (oliveras@lsi.upc.edu) Date: Fri Jul 21 09:42:26 2006 Subject: [SMTCOMP] a question on the benchmark scrambler In-Reply-To: References: <200607211559.43967.alberto.griggio@dit.unitn.it> Message-ID: <1116.81.38.209.206.1153500135.squirrel@webmail.lsi.upc.edu> Hi all, you are right, BCTL depends on this restriction. As Leonardo pointed, I took that assumption from the SMT-LIB specification. Cheers Albert > Hi Roberto, > >> I suspect barcelogic does depends on this restrictio. Am I wrong? > > I don't know which tools depend on this restriction, but > I don't see a problem with them since the SMT-LIB specifies this > restriction. > > Leonardo > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.csl.sri.com/pipermail/smtcomp/attachments/20060721/13e48adf/attachment.html From demoura at csl.sri.com Tue Jul 25 12:26:08 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Tue Jul 25 12:26:11 2006 Subject: [SMTCOMP] SMT-COMP info Message-ID: <40DD41D5-1F1C-44A3-A658-E16C6043A58F@csl.sri.com> Hi, The deadline for submitting solvers is August 8. We will *not* extend the deadline this year. The organizers will post their solvers on August 7 before the other participants submit the magic numbers for the pseudo-random selection of benchmarks. We updated the benchmarks in the SMT-COMP and SMT-LIB websites. They are synchronized now. The new benchmarks contain minor modifications: - broke egt family in QF_UFBV32. - removed nonlinear (simplify) benchmark from AUFLIA. - removed one non QF_UFIDL benchmark (UCLID-pred/BRP/BRP0.smt) - replaced uninterpreted sort with Int sort in a couple of benchmarks in QF_AUFLIA/swap. All benchmarks were validated, and this is the official benchmark library for SMT-COMP'06. Best wishes, Leonardo for Aaron and Clark too. From cimatti at itc.it Tue Jul 25 22:31:31 2006 From: cimatti at itc.it (Alessandro Cimatti) Date: Tue Jul 25 22:40:58 2006 Subject: [SMTCOMP] RE: QF_UFBV: on last-minute extension with sign In-Reply-To: <200607111554.k6BFsmkx005011@zaphod.cs.nyu.edu> Message-ID: Hi Clark, the new version of the QF_UFBV benchmarks is indeed fixed - there are no undefined symbols - and they are an excellent addition to SMT-LIB. The use of signed operators is a not a major extension, and it seems that many of the interesting benchmarks make use of them. So, I now think that they shuold all be included in the competition. Thanks for your efforts on setting everything up, Alessandro > -----Original Message----- > From: Clark Barrett [mailto:barrett@cs.nyu.edu] > Sent: Tuesday, July 11, 2006 5:55 PM > To: Alessandro Cimatti > Cc: Clark Barrett; SMT-LIB; smtcomp@csl.sri.com > Subject: Re: QF_UFBV: on last-minute extension with sign > > > However, I think that the extension of QF_UFBV32 with signed operators > > should not be accepted at this stage of the competition. > > > > In fact, the bit vectors theory has been fixed a long time ago --- the > > description was posted on the list by Cesare on May 5th. The > > description made it clear that the interpretation would be > > unsigned. That signed arithmetic would not be part of the theory had > > been discussed and agreed upon with Cesare, Silvio, and others. > > > > No variations were communicated on the topic since. Thus, the > > competitors entering this category had no reason to include signed > > operators. A last minute addition to the theory would give them a > > disadvantage against those who have signed arithmetic operators > > already available, or somehow knew that the extension was being taken > > into consideration. > > > > Notice that this is really a last minute extension: for the other > > categories, not only the theories, but also the benchmarks, have been > > frozen more than a week ago. > > Let me comment on why this happened. When we first formalized the bit- > vector > theory, we anticipated receiving benchmarks from industry that would fit > nicely > in the defined theory and logic. Unfortunately, these benchmarks never > came. > When it became clear they would not be available, I went looking for other > benchmarks. Stanford offered the ones that I ultimately translated. > Unfortunately, most of them required the use of signed arithmetic as well > as > arrays. > > One important aim of SMT-LIB is to capture benchmarks in their native form > and > avoid as much as possible translations and re-interpretations, so I was > reluctant to make any changes to the theories and signatures used by the > benchmarks. > > In fact, I was contemplating adding a theory combining bitvectors with > arrays > to properly translate the benchmarks, but upon closer examination, I > realized > that in most cases, arrays were simply being used like functions (i.e. the > update/store/write feature was not being used). Thus, I felt that it was > more > accurate to represent these as uninterpreted functions and I made that > translation. > > A similar translation could have been done to eliminate the signed > arithmetic > operations, but I felt that would be against the spirit of SMT-LIB because > these are common bit-vector operations, and they should be included in the > logic instead. I am happy with the benchmarks as they are and feel that > they > are an accurate representation of the original application and a good > addition > to SMT-LIB. > > The question of whether they should be included in SMT-COMP is somewhat > orthogonal, but related. I tend to think they should be included for the > following reasons: > > 1) First, let me clear up a misunderstanding resulting from a typo in the > previous version of the logic (see my previous email). It is actually > the > case that 6819 of 8247 files (83%) use the signed operators, not just > 437. > Perhaps there is still enough variation in the remaining benchmarks, > but I > don't know. By the way, I suspect this also explains Alessandro's note > about undeclared symbols. > > 2) The signed comparison and extend operators can be expressed fairly > simply in > terms of the previously existing operators, so it shouldn't be > difficult to > add the extensions. > > 3) One possible criticism is that "native" support for signed operators > will be > more efficient than implementing them as existing operators, but for my > own > part, I can say that I expect my system would have the same performance > and > I don't know of any system that has special performance capabilities > for > signed operations. > > That said, I am interested in what others think. Should we use the full > set of > benchmarks for SMT-COMP, or restrict it to the 17% of them that do not use > the > signed operators? I suggest that the remainder of this discussion take > place > on the smt-comp list rather than the smt-lib list. > > Clark Barrett From rseba at dit.unitn.it Wed Jul 26 00:38:22 2006 From: rseba at dit.unitn.it (Roberto Sebastiani) Date: Wed Jul 26 00:38:34 2006 Subject: [SMTCOMP] CFP: JSAT S.I. on Satisfiability Modulo Theories Message-ID: <200607260738.k6Q7cMIL022214@dit.unitn.it> [We apologize if you receive multiple copies of this announcement] =================== Preliminary Call for Papers ==================== Journal on Satisfiability, Boolean Modeling and Computation (JSAT) Special Issue on Satisfiability Modulo Theories (SMT) http://dit.unitn.it/~rseba/jsat_smt06/ Deadline for paper submission (provisional): November 25th, 2006 ====================================================================== GENERAL INFORMATION Satisfiability Modulo Theories (SMT) is the problem of deciding the satisfiability of first-order formulae with respect to some decidable background theory (e.g., linear arithmetic, the theory of arrays, the theory of bit-vectors). SMT techniques are gaining increasing relevance in many application domains, including formal verification of hardware and software, compiler optimization, planning and scheduling. SMT is strongly related to SAT, as most SMT tools are built on top of or interface with efficient SAT solvers. (See also the SMT-LIB page http://combination.cs.uiowa.edu/smtlib/.) TOPICS Topics of interested include, but are not restricted to: * Novel general SMT techniques * Novel SMT techniques for theories of interest * SMT for combined theories * Novel implementation techniques for SMT * Applications of SMT SUBMISSIONS This special issue welcomes original high-quality contributions that have been neither published in nor submitted to any journals or refereed conferences. All submissions should be written in terms understandable by general readers of the journal. All submissions will be refereed according to JSAT standards, as described at JSAT web page (see below). Submissions should be written in LaTeX and formatted according to JSAT's author guidelines, and should not exceed 20 pages. JSAT LaTeX style file can be obtained at the journal web page (see below). ABOUT JSAT JSAT (http://www.isa.ewi.tudelft.nl/Jsat/) is a peer-reviewed journal which is freely distributed electronically and published in print by IOS Press. The scope of JSAT is propositional reasoning, modeling and computation, and related topics. JSAT publishes high-quality original research papers and survey papers which evidently contribute to deeper insight on a SAT-related topic. GUEST EDITORS Byron Cook, Microsoft Research - bycook@microsoft.com Roberto Sebastiani, DIT, Universita` di Trento - rseba@dit.unitn.it FURTHER INFORMATION For further information visit the web page: http://dit.unitn.it/~rseba/jsat_smt06/ or send an email to one of the guest editors. From rseba at dit.unitn.it Wed Jul 26 00:24:57 2006 From: rseba at dit.unitn.it (Roberto Sebastiani) Date: Wed Jul 26 01:47:15 2006 Subject: [SMTCOMP] CFP: JSAT S.I. on Satisfiability Modulo Theories Message-ID: <200607260724.k6Q7OvVZ021223@dit.unitn.it> [We apologize if you receive multiple copies of this announcement] =================== Preliminary Call for Papers ==================== Journal on Satisfiability, Boolean Modeling and Computation (JSAT) Special Issue on Satisfiability Modulo Theories (SMT) http://dit.unitn.it/~rseba/jsat_smt06/ Deadline for paper submission (provisional): November 25th, 2006 ====================================================================== GENERAL INFORMATION Satisfiability Modulo Theories (SMT) is the problem of deciding the satisfiability of first-order formulae with respect to some decidable background theory (e.g., linear arithmetic, the theory of arrays, the theory of bit-vectors). SMT techniques are gaining increasing relevance in many application domains, including formal verification of hardware and software, compiler optimization, planning and scheduling. SMT is strongly related to SAT, as most SMT tools are built on top of or interface with efficient SAT solvers. (See also the SMT-LIB page http://combination.cs.uiowa.edu/smtlib/.) TOPICS Topics of interested include, but are not restricted to: * Novel general SMT techniques * Novel SMT techniques for theories of interest * SMT for combined theories * Novel implementation techniques for SMT * Applications of SMT SUBMISSIONS This special issue welcomes original high-quality contributions that have been neither published in nor submitted to any journals or refereed conferences. All submissions should be written in terms understandable by general readers of the journal. All submissions will be refereed according to JSAT standards, as described at JSAT web page (see below). Submissions should be written in LaTeX and formatted according to JSAT's author guidelines, and should not exceed 20 pages. JSAT LaTeX style file can be obtained at the journal web page (see below). ABOUT JSAT JSAT (http://www.isa.ewi.tudelft.nl/Jsat/) is a peer-reviewed journal which is freely distributed electronically and published in print by IOS Press. The scope of JSAT is propositional reasoning, modeling and computation, and related topics. JSAT publishes high-quality original research papers and survey papers which evidently contribute to deeper insight on a SAT-related topic. GUEST EDITORS Byron Cook, Microsoft Research - bycook@microsoft.com Roberto Sebastiani, DIT, Universita` di Trento - rseba@dit.unitn.it FURTHER INFORMATION For further information visit the web page: http://dit.unitn.it/~rseba/jsat_smt06/ or send an email to one of the guest editors. From demoura at csl.sri.com Wed Jul 26 09:23:44 2006 From: demoura at csl.sri.com (Leonardo de Moura) Date: Wed Jul 26 09:23:47 2006 Subject: [SMTCOMP] degree of difficulty in QF_UFBV[32] Message-ID: <0B5EFDB7-8834-4359-94E6-E9BB8B5306A9@csl.sri.com> Hi all, As you probably know, the SMT-COMP website describes the approach used to compute how "difficult" a benchmark is. (http://www.csl.sri.com/users/demoura/smt-comp/bench_selection.shtml). However, we used a different approach for QF_UFBV[32], because none of the solvers listed on the website supported the bitvector theory and/or a parser for the new SMT-LIB extensions. Our first idea was to assign degree 5 to everything, but this would be bad because there are several easy benchmarks, and we could end with a trivial division. So, we decided to use a recent version of CVC Lite to compute the difficulty levels. The degree of difficulty was obtained using the following table: Time Lvl 0 : 0 < 5 : 1 < 10 : 2 < 100 : 3 < 1000 : 4 timeout : 5 Note this is not nepotism. This alternative approach does *not* give an advantage to CVC Lite. Actually, it is bad for CVC Lite because half of the benchmarks selected for SMT-COMP will be hard for CVC Lite. Leonardo