MAPCAR of DESTRUCTURING-BIND in DEFUN throws error evaluating the defun

[nigel-me]

Found this when destructuring some data.

Corman Lisp gives error (when SBCL doesn't) when mapcar and destructuring-bind are used together in a defun
but fine to have 1) destructuring-bind in defun
and 2) destructuring-bind in mapcar
and 3) putting destructuring-bind in a helper function that mapcar calls in defun works
but both in defun doesn't:

(defparameter data '((a (b c d) . e) (1 (2 3 4) . 5)))
DATA

(defun extract-scale-scores (mydata)
"Extract scale names and their floating-point scores from the given data."
(destructuring-bind (a (&rest b) . c) mydata (cons a c)))
;;; Warning: Unused variable B in function EXTRACT-SCALE-SCORES
EXTRACT-SCALE-SCORES
(extract-scale-scores (first data))
(A . E)
(mapcar #'(lambda (zz) (destructuring-bind (a (&rest b) . c) zz (cons a c))) data)
;;; Warning: Unused variable B in anonymous function
((A . E) (1 . 5))
(defun extract-it (mystuff) (mapcar #'(lambda (zz) (destructuring-bind (a (&rest b) . c) zz (cons a c))) mystuff))
;;; An error of type TYPE-ERROR was detected in function SIGNAL-TYPE-ERROR:
;;; Error: Type error: datum = C, expected type = LIST
;;; Entering Corman Lisp debug loop.
;;; Use :C followed by an option to exit. Type :HELP for help.
;;; Restart options:
;;; 1 Abort to top level.
:C 1

;;; Returning to top level loop.
(defun extract-it (mystuff) (mapcar #'extract-scale-scores mystuff))
EXTRACT-IT
data
((A (B C D) . E) (1 (2 3 4) . 5))
(extract-it data)
((A . E) (1 . 5))

combined defun, mapcar, destructuring-bind works in SBCL (portacle):
CL-USER> (defun extract-it (mystuff) (mapcar #'(lambda (zz) (destructuring-bind (a (&rest b) . c) zz (cons a c))) mystuff))
; in: DEFUN EXTRACT-IT
; (DESTRUCTURING-BIND (A (&REST B) . C) ZZ (CONS A C))
; --> SB-INT:BINDING*
; ==>
; (LET* ((#:G8 (SB-C::CHECK-DS-LIST/&REST ZZ 2 2 '(A # . C)))
; (A (POP #:G8))
; (#:G9 (POP #:G8))
; (B #:G9)
; (C #:G8))
; (CONS A C))
;
; caught STYLE-WARNING:
; The variable B is defined but never used.
;
; compilation unit finished
; caught 1 STYLE-WARNING condition
EXTRACT-IT
CL-USER> (defparameter data '((a (b c d) . e) (1 (2 3 4) . 5)))
DATA
CL-USER> data
((A (B C D) . E) (1 (2 3 4) . 5))
CL-USER> (extract-it data)
((A . E) (1 . 5))
CL-USER>

Also works in CLISP like SBCL

PS editor sees ear-muffs as markup in DEFPARAMETER etc.

[j3pic]

The problem is due to the function COMMON-LISP::TRAVERSE-FORM, which attempts to look for RETURN and RETURN-FROM forms.

It seems to assume that the form it will be passed will not contain any unexpanded macros. For whatever reason, this condition holds only as long as there are no lambda forms within the defun form. If there are any lambda forms, then any use of dot notation within the syntax of a macro will fail with the error above. For example, this defun fails:

(defun dot-notation-bad ()
  (lambda ()
    `(1 . 2)))

If you replace the backquote in the example above with a normal quote, it will succeed, but that's only because there is a special case in common-lisp::traverse-form for quote.

If you remove the lambda form, the backquote will be macroexpanded before the call to common-lisp::traverse-form:

(defun dot-notation-okay ()
  `(1 . 2))

This will also fail:

(defun dot-notation-bad ()
  (lambda (&optional (zz `(1 . 2)))
    nil))

Fixing this bug correctly requires understanding why the bodies of lambdas are shielded from macroexpansion. Either they should be macroexpanded too, or whatever traverse-form is trying to accomplish should be either removed or implemented some other way.

It would take some research to figure any of that out. Unfortunately, MACROEXPAND-1 and MACROEXPAND are protected from tracing in Corman Lisp, and the IDE won't let me jump to their definitions (meaning they're probably written in C++). I have no C++ environment for Windows.

[j3pic]

The following workaround avoids the error:

(setf common-lisp::*optimize-tail-recursion* nil)

(defun dot-notation-okay-now ()
  (lambda (&optional (zz `(1 . 2)))
    nil))