Hi everyone,
I put out TXR 92 today.
Binaries are available from SourceForge.
See the new RELNOTES document.
http://www.kylheku.com/cgit/txr/plain/RELNOTES
Txr now runs on Cygwin.
Some significant bugs are fixed, including a spectacular bug in function calling that dates back to 2009.
There are a bunch of new features.
- @(skip) has a :greedy keyword, which extends its usefulness
- @(trailer) and @(skip) work horizontally now too
- collection directives have new keywords to control what
variables are collected and a minimum and maximum number
of times to collect.
- new @(eol) directive matches end of line explicitly.
You can do useful things like
@(collect :mintimes 3 :vars (a b (c "foo")))
which means collect at least three times (or else fail), and only the variables a, b and c are collected. For any successful collect iteration, if a binding for c is not produced, then a binding of "foo" is substituted.
Match last three white-space-delimited tokens in a line:
@(skip :greedy) @a @b @c
Match trailing context in a line. Collect space-delimited tokens into a, which are not followed by a number. Then collect the rest.
query:
@(coll :gap 0)@a @(trailer)@/[^0-9]/@(end)@rest
input:
a b c 1 2 3
output:
a[0]="a"
a[1]="b"
rest="c 1 2 3"
Grab the first and last character of the line:
@{first 1}@(skip :greedy)@{last 1}
Cheers ...
Sorry for neglecting to post news! :)
See the ChangeLog to see what's been done in 036, 037 and 038.
Bugfixes.
Since version 028, there has have been a few GC fixes, a bunch of regex work. Check it out!
I implemented regex derivatives (in addition to the existing NFA impelmentation, not as a replacement). Txr now has some powerful regex operators: intersection (&), complement (~) as well as a non-greedy closure (%).
While working on this, I found some problems in the regex syntax and fixed them.
I've done some code cleanup and fixed one GC-related bug.
The installation prefix now uses the widespread DESTDIR convention. E.g. you can configure with --prefix=/usr, but then do a "make install DESTDIR=temp", which will install everything into the directory temp/usr, from which you can roll up a package. That's how people prepare packages GNU/Linux distros. If you use the --install_prefix configure option, it sets up DESTDIR.
After "make install" you can now do "make install-tests" which will add all of the test cases to the installation, under the usr/share/txr/tests. This "make install-tests" also recursively invokes "make -sn tests" and captures the commands which execute the test cases; these are spun into a script called run.sh. Currently, the run.sh script requires write access to the directory where the test cases are installed. Ah well; it's a development feature.
make install-tests makes it easy to run the same test as "make tests" but on a cross target system. That's how I found the gc problem; I cross-compiled txr for a MIPS system, made the cross test package, and ran it on a MIPS board.
I introduced Valgrind integration in the garbage collector. You must pass the --valgrind option to the configure script to set this up. The basic support squashes false positive error reports from valgrind that occur during the marking phase of the mark-and-sweep, since the stack scanning can touch uninitialized data. Furthermore, the txr option --vg-debug, turns on more advanced valgrind support: objects on the GC free/recycle list are marked as inaccessible such that accesses to free objects, while they are free, will be trapped by valgrind. In this mode, FIFO recycling is used: least recently freed objects are allocated first, to maximize the time that a reclaimed object stays on the free list, thus maximizing the chances of catching an invalid ccess to it. One other Valgrind-related fix is to suppress some false positive leak reports: txr's vectors are stored in such a way that the vector data is referenced by an internal pointer, not by a pointer to the base. To valgrind, this looks suspicious. So when Valgrind support is enabled, the library maintains an additional pointer, so the debugger knows that the objects are genuinely reachable.
I fixed some build issues and added configure-time detection for yacc.
Found and fixed two cases of uninitialized memory. (Amazing what you find when valgrind doesn't flood you with false positives!)
The test cases run with zero reports from Valgrind now. (Disclaimer: on my machine, etc). :)
In the @(next) and @(output) directive, the args and nothrow keywords are really keyword symbols now: have to be written as :nothrow and :args. This will be instantly recognizeable by those of you who are Lispers as making sense.
I made enhancements to the configure script, which now autodetects some information by compiling sample test programs. So for instance, the integer type which converts to and from pointers is not just assumed to be "long", but is auto-detected.
Symbols are now maintained in hash-table based packages instead of in a global association lists. There are three packages: system, user and keywords.
Keywords always stand for themselves. The documentation also makes it clear that the symbols nil and t are special also and stand for themselves.
The behavior of the program is cleaned up in this regard too; @(bind) had some bugs. Now for instance @(bind nil t) behaves as a failed match, @(bind :foo :foo) is a successful match (keyword matches itself), etc.
A bug in the @(flatten) directive is gone (the underlying function was reducing an empty list to a list containing an empty list, following the intended pattern atom -> (atom), which should not apply to the empty list atom. This kind of thing is waht you get when you follow the beautiful design pattern for lists from Lisp.
Plus other bugfixes, like the broken handling in the parser and such, and lots of internal improvements in the code base.
I fixed an embarrassing bug in yyerror and yyerrorf. The problem is that all of the current test cases use correct queries; the handling of syntax errors is not exercised. The test has a coverage which is too poor to be useful for the discovery of regressions in the face of broadly sweeping changes throughout the code base.
Also, a @(collect) and @(coll) bug that was introduced in txr-015 is now fixed. What was happening is that whenever a collect didn't yield any bindings (even if its body otherwise matched something one or more times!) it would generate a match failure.
The program is now purged of dependencies on GNU extensions and C99 features. All UTF-8 handling is done without dependencies on wide character stream I/O the C library. The program now requires only C90, some <wchar.h> functions that were added to C in the 1995 addendum, and functions from the POSIX.1 and POSIX.2 API's.
Fixed the output string stream crash, bug 28033.
There is a new kernel object type (identified by a 11 bit pattern in the tag). This represents C strings, and allows string literals to be first-class objects of string type, without consing any additional memory. If one is careful, this type can also be used to smuggle other C strings into functions, provided that these uses do not ``escape'' the context in which they are used. The C strings used this way have to be aligned to at least a four byte boundary: don't use with substrings!
First class literals mean that functions no longer have to use raw C strings in order to be used with literals. So for example format string in the format function now is ``obj_t '' and now ``wchar_t ''. You can pass a literal object, using the lit macro: lit("foo ~a") or you can pass a string object, and even a lazy string.
I've also eliminated C printf style formatting from the streams interface, and from all functions built on top of streams, like formatted exception throwing functions and such.
I've converted uses of raw C streams to using the object streams; e.g. where there was a call to fwprintf(stdout, ...), there is now a format(std_output, ...). The dependency on C streams is now minimal and confined to the streams module.
The next project task will be to eliminate the use of C99 wide streams, doing all UTF-8 encoding and decoding using the UTF8 module.
The main feature is Unicode/UTF-8 support. Queries and data are UTF-8 now. Regexes support Unicode.
Termination status of command pipes is now processed and turned into an exception.
There are bugfixes, and internal improvements.
A new hash table type is provided internally but not yet put to any use.
The (obj_t *) value/reference type now can hold character and integer values right inside the pointer. The low two bits are used as a tag. Thus operations which manipulate the CHR and NUM types no longer have the overhead of consing up memory and generating garbage. Txr runs quite a bit faster as a result, depending on the kind of query.
There is now an autoconf-like build configure script, but without the bloat and braindamage of actually using Autoconf.
There is cross-compiling support, and support for building and running tests in separate directory. (But not yet any support for generating a test package to be run on cross target when cross-compiling).
I made some improvements to the man page, and fixed three bugs: incorrect test case 002, and two bugs that caused txr to crash or abort on x86-64.
A new freeform directive has been added.
Regexps can be bound to variables. Matching with variables bound to regexps uses the regexp.
This has some important bugfixes:
The @(skip) directive was discovered not to be yielding the correct line past the material that was matched, allowing subsequent query forms to go over the same material.
@(collect) was discovered to loop infinitely in the case that the collect body succeeds without matching anything (and yields the same match position); it now moves by one line in that situation.
A new test case 004 was added to guard against regression in these behaviors.