Exploring the J Language and Array Programming

Why Array Programming?

I've been developing software in Clojure since 2010. It is the language that I know most deeply and which has affected my technical development and thinking more than any other.

In 2021 I made a career shift into engineering management. My role as an engineering manager has broadened my horizons, including the use-cases for which I write code. I now find myself in roles on the other side of the software engineering table: analyst, stakeholder, subject matter expert, and evaluator of systems both human and digital.

My programming tasks are now centered around data analysis, and this has increased the usefulness of array programming languages for doing my job.

I initially stumbled on the J language in 2017, and saw it as another challenge, a leap into an area of computer science that I had not yet traversed. A little over a year ago, I had opportunity to start listening to the ArrayCast podcast during a personal period filled with an unusal amount of daily commuting by car. Covering general array language topics not only in APL but across the languages inspired by it, these podcast discussions about the history, usage, and underlying concepts of these "Iversonian" languages have helped (and continue to help!) solidify my understanding of them and inspire me to continue learning more about this paradigm for solving problems with the computer.

Some of this learning has been array language-specific; other parts have been filling holes in my mathematical education that have long needed filling. They are all about solving real-world, immediate, important-to-me and important-to-my-employer problems, answering concrete questions, validating urgent hypotheses. As such, I find myself drawn to these observations of Roger Hui and Morten Kromberg regarding APL and its early success:¹

...APL on the mainframe was a highly productive environment which allowed engineers, actuaries, and managers to write models and crunch numbers. APL users had efficient access to data in all the various file formats that the mainframe had to offer, state-of-the-art systems for producing 24×80 “fullscreen” user interfaces, and budding business graphics systems...

For software engineers who trained for years to do refactoring, this was an exciting time. But for someone who wanted to solve problems with a computer and had been able to do it easily in APL on the mainframe, it was a disaster. The highly productive environment in which smart people who had other primary skills than programming–such as bond trading or chemical engineering—could build reliable multi-user applications, was replaced by a jumble of unreliable, rapidly evolving APIs created for software engineers.

The mention of "rapidly evolving APIs" leads to one of my first fundamental observations about the Iversonian languages, which I share in the next section.

Fewer Branches of Mathematics

As I've spent more time recently both learning and practicing array programming in J, APL, BQN, and K, I believe the following to be true:

Every software program defines a new branch of mathematics.

By mathematics, I mean as Keith Devlin has put it, the science of patterns.

In particular, I've realized that each program written in non-Iversonian languages establishes a new branch of mathematics from an extremely low level of mathematical abstraction. Part of the joy, but also the toil, of writing in those languages is establishing the primitives of your new, tiny branch of mathematics. Building a mathematical language that harmonizes well with the source language, the machine, and which also composes effectively with itself is a non-trivial feat of design and foresight.

Part of my journey into array languages has been to let go of wanting to develop my own mathematics, and instead to embrace the wealth of relationships and patterns that are inherent in traditional mathematics and which are grafted directly, exposed unabashedly in Iversonian programming languages.

The promise of array languages is this: the core primitives of these languages are general enough that solving problems with them involves thoughtful composition of the primitives, rather than needing to forge new ones.

Some programs grow to a size that demands further fundamental abstractions be developed. However, our intuition for that necessary complexity threshold has been biased by popular programming languages and their modes of expression. Arthur Whitney once demonstrated a graphical text editor written in the K language that fit on four lines of code that could be read in their entirety in a single visual span².

Leveraging these mathematical patterns with arrays as the central data structure leads to largely loopless and branchless programs, in which algorithms are distilled to a series of progressive array transformations.

There's no free lunch. Not only are the patterns different from mainstream programming languages, but they have different trade-offs in terms of time and space complexity. These languages are interpreted, which demands different optimizations, and the implementation of production-grade interpreters for array programming languages is non-trivial.

Which array programming language?

In 2017, I only gave J the time of day. I had so much to learn that I could not explore more than one of these array languages at once.

Over the last year I've spent more time learning APL, BQN, and K.³ When I was first learning Clojure, I recalled a phase where studying other functional programming languages helped solidify for me what was common to functional programming as a paradigm, and what was unique to Clojure as a Lisp on the JVM. The same has been true while diving into these other array languages.

There are a number of superficial differences between these languages, as well as a number of significant differences that can be difficult to perceive or understand at the outset. If you've never picked up an array language, picking one of APL, BQN, or J will give you a taste for the paradigm. K is a different beast, worthy of independent consideration in my opinion.

As of January 2024, there are several active communities online for discussing array programming, including The APL Farm on Discord and Matrix.

J will remain my go-to array programming language for now for these reasons:

• Open source
• Core language distribution (with addons) largely handles development essentials
• Good performance
• Mature, tested implementation
• A lot of freely-available and interesting educational material

The BQN language exhibits many of these same virtues and provides compelling alternatives to some of the language design decisions in J and APL. As a language, BQN also involves less manipulation of programs as strings and provides more structured, functional facilities like its blocks. I look forward to the maturation of the BQN ecosystem and will be keeping an eye on its development.

All of the strengths of array programming aside, asking other people who have no experience with array programming to read J or BQN or APL code and assess the results obtained from it would be like asking an English speaker to assess a conversation in Russian after a 30-minute tutorial on the pronunciation and grammar of the language. Other tools are needed when work must be shared, data stories told, and coworkers' feedback solicited.

For sharing data analysis work, I've been studying and practicing with Julia, but that's a topic for another post.

If both content and time permit, I hope to share specific things that I have learned and will learn about J and array programming. You can subscribe to the array-programming RSS feed on this site and you can view the content I publish on the J wiki.

Tags: kenneth-iverson jsoftware k-language clojure j-language apl bqn array-programming exploring-j

K Language: Upper Case

Julia's Error Message for String Concatenation