-- mode: org; --

#+HTML_HEAD: #+HTML_HEAD:

#+HTML_HEAD: #+HTML_HEAD: #+HTML_HEAD: #+HTML_HEAD:

This is a template for pushing out org files that are compatible

with both HTML and latex export. Specifically, these files

Support code highlighting - for clojure code - and typeset

the code to look different from the main sections. The

design is meant for providing a quick template to inject

clojure source into org docs, and provide an interactive,

REPL-friendly presentation.

#+TITLE: CLClojure: An Experiment Port of Clojure to Common Lisp #+AUTHOR: joinr #+DATE: 18 Aug 2018

• Background Porting Clojure seems to be the thing to do these days. After the clojurescript compiler came out, people had a really good blueprint for hosting clojure in other languages. The fact that the clojurescript compiler is defined, largely, in terms of protocols, makes it a bit closer to the "clojure in clojure" goal. As a result, you should be able to bootstrap the language (as they did with JavaScript) with a minimum of primitives. Most of the language is defined in Clojure, so voila.

We currently have Clojure ports targetting Scheme, C, C-via-Scheme, Python, etc. I've found Common Lisp to be a particularly slick environment with some cool tools and a fairly active community. I think it would be interesting to port the ideas from Clojure to Common Lisp, ultimately implementing a compliant Clojure in Common Lisp.

• Layout The current WIP is broken into several Common Lisp files (only tested on SBCL). You can look at the dependencies in clclojure.asd. The general layout and organization follows (subject to change): ** common-utils Contains a plethora of utility functions and convenient clojure analogues (or direct ports) useful for porting clojure source. Primary features include multi-arity lambdas via =lambda*= and multi-arity functions via =defun*=. Also provides a =with-recur= form that allows for a similar idiom for Clojure's =loop/recur=. This is typically pulled in by all other packages for general utility. Some limited sequence functionality overlaps with sequences.lisp, and may be replaced in the future. ** sequences Provides an implementation of clojure's lazy sequence abstraction. Implemented in CLOS, and porting the seq protocols. Provides a subset of the clojure core sequence library for CL used in bootstrapping and metaprogramming. The intent is not to have a full port, but a minimally useful subset. Integrates the seq abstraction with CLOS sequence types. ** pvector A port of the java implementation of clojure's persistent vector based on Hash Array Mapped Tries (HAMTs). Serves as a stand-alone datastructure with local functions. Also defines =subvec= which is the O(1) slice of a persistent vector. ** cowmap A simple persistent map implemented by copy-on-write operations over a standard hash table. This is a minimal implementation to support bootstrapping, and not intended for performance or production use. ** pmap (wip) A port of the java implementation of clojure's persistent hashmap. Incomplete. ** protocols Defines forms for =defprotocol= and =deftype= in common lisp. Protocols are implemented as bundles of generic functions and args, stored in a CLOS struct. Protocols are registered, and define =satisfies?= predicates for the protocol to ensure valid implementations of =reify= =deftype= =extend-type= and =extend-protocol=. Protocol definitions expect vector args (rewrite pending to extend these to sequences to allow simple cl-based protocol definitions).

** reader Provides a consolidated place for reader macro support. Primary purpose is to enable vectors, maps, and sets for bootstrapping. Original experiments also supported characters. Intentionally hijacks the read-table. Needs to be ported to named-readtables, or excluded entirely once bootstrapping of clojure.tools.reader is done.

** eval Due to the custom evaluation rules for clojure's data literals, simple reader macros are not enough to get literal support. This package defines a generic function, =custom-eval=, and an API for installing/uninstalling custom evaluation support in SBCL. The implementation swaps the function definition for =SB-IMPL::simple-eval-in-lexenv= and adds a final condition to the non-list eval rules: if =custom-eval= is supported, then the generic function is invoked. This allows user-defined extension of the evaluation semantics in a minimal way, and retains low-level support (e.g. from the compiler).

** literals Consolidated place to extend custom evaluation rules for vectors, maps, and other literals. ** keywordfunc Initial hack at implementing keywords-as-functions support in CL. This currently works only in lexical forms, and is not a global modification. Exposes the macro =with-key-fn= which will scan the body for keywords in function position, and build a table of keyword functions and fbind. The more commonly used feature is the =keyaccess= type, which is used in the lexical package. ** lexical Implements a lexical environment ala =let*= called =unified-let*= that will unify lisp-2 bindings into a lisp-1 lexical environment for the body. This is the foundational method for implementing clojure-style =let=. ** bootstrap Implements a core set of clojure special forms and brings along the protocol definitions from cljs.core. Eventually, the entire cljs.core library will be pulled in here and evaluated. This should hopefully provide a sufficiently robust environment to then bootstrap the rest of clojure via tools.reader and tools.analyzer, leveraging extant CL implementations from the earlier bootstrapping process where it makes sense. ** symbols (wip) Nascent thought piece on implement clojure-specific vars and namespaces. Clojure's qualified symbols and keywords are a tad different and require some thought towards integration with CL.

• Goals ** Bridge the gap between the cool stuff in clojure and Common Lisp.

** Implement really useful bits of clojure in portable common lisp, and provide them as stand-alone libraries.

This includes lazy sequences, the generic sequence abstraction, and the fundamental persistent hash-array-mapped-trie data structures in clojure:

• persistent vectors
• persistent maps
• persistent sets.

** Extend the generic sequence abstraction and other idioms to Common Lisp's built-in mutable structures.
Common Lisp already has a sequence library, but I think Clojure's is more general and can be trivially extended to new types.
Common Lisp's irrational locking down of SEQUENCE is a hurdle here. The HYPERSPEC will never be updated in my lifetime :) So generic functions are the current way to bridge this stuff.

• Protocols are really nice, as are Clojure's arbitrary dispatch multimethods.
• Data literals are also highly useful. I think Clojure nailed the choice of literals, so providing reader macros for these guys would be very nice.

** Possibly wrap a Common Lisp STM implementation, or cheat and use something like lparallel or just delegate to clojure.core.async (like clojurescript).

** Bootstrap a fully functional Clojure onto Common Lisp.

** Learn. So far, this project continues to be a great way to learn about both CL and Clojure, to include implementation hurdles, support for PLT stuff, long-standing warts from decisions made ~25 years or more ago, and work-arounds. Nothing is insurmountable so far, which is pretty cool.

• Status ** Began work porting Clojure's persistent data structures from Java about years ago, while simultaneously learning Common Lisp (and by necessity Java :/ ).

• Got a working persistent vector implementation, with compatible clojure literals about a year ago.

• Started working on persistent hash maps around November 2012.

• Built a temporary implementation of clojure-style protocols in Common Lisp ~ Dec 2012.

• Pulled the bits into an actual ASDF system and pushed everything to the Github August 2013.

• Implemented a baby clojure evaluator that should bridge the lisp1/lisp2 gap between clojure and the Common Lisp host. Unsure if this is going to work out in the long term, but eh.

• It's real trivial at the moment.

• Working on library code in my spare time, still need to port the persistent map.

** Revisited 2017

• implemented some rudimentary stuff
• vector reader macros not fleshed out; work fine at the REPL, but failed to return a vector (instead returning a form to create the vector). Trivial but important oversight.
• Still hijacking the global read-table. Pulled in named-readtables to help, but haven't integrated.
• Working on variadic functions, explored funcallable classes, refined protocols, deftype.
• cleaned up the build process (more work to be done here)

** Revisiting 2018

• got reader macros matured (vector literals produced properly now),

• got protocol definitions and implementations respecting vectors, albeit in a hacky way. We still allow lists for args....

• working on deftype, then getting the bootstrap from CLJS (core protocols and functions) to compile.

• Working on leveraging named-readtables for better delineation of clojure source, to include file-level (somewhat racket like).

• Also intend to leverage named-readtable to get case-senstive reader (via :modern), and to enable CL interop trivially with a macro allows CL reader inside body (vice versa, for inline clojure if it makes sense).

• refining the approach after reading a lot, looking at some other sources of inspiration (including early proto-clojure->java compiler from Hickey in CL)

• Basic def, fn works. Protocols work. Metadata works mostly. Deftype

• got let, albeit without destructuring. Still useful for bootstrapping!

• Initial implementation of reify working, wrapped deftype in a version compatible with cl:deftype

** Revisiting 2019 *** Reconciled some problems with data literals, clojure's evaluation model, and CL's model. The issue is tricky and - even after using Clojure for 8+ years - something I completely missed.

So folks laud CL having awesome reader macros, and talking about how they allow you to implement any language etc. Banking on these comments, I went about doing that (Clojure). As it turns out....IF you want to define a language with new data literals , and those literals have their OWN evaluation semantics, reader macros cannot help you (they help to a point).

• I think racket probably better supports this based on research, but I have no proof.

When we read a vector in clojure (not a syntax-quoted one), we actually get a literal vector data structure (not a a list of `(vector arg1 arg2 arg3), but an actual vector of [(eval arg1) (eval arg2) ...]).

I chased my tail on this twice now, and finally realized that in clojure, (eval [x y]) -> [(eval 'x) (eval 'y)].

So when you read a vector, and eval it, you actually have to apply those evaluation semantics to build up the resulting vector. In CL, eval just passes the vector through.

This poses a problem in CL, since yes, you can trivially extend the reader to handle vectors, but you CANNOT get the evaluation semantics to line up, since CL provides no mechanism to do so (unlike say reader macros). You can of course define your whole infrastructure, including your own eval, etc., but I'm interested in minimally boots