Robin Sloan
The Media Lab
September 2021

Fantasy query languages

A fragmentary papyrus scroll, obviously ancient, with a several straight lines of Egyptian numbers.
Papyrus inscribed with accounts of beer and bulls, ca. 1981–1975 B.C., Middle Kingdom

This is where I’ll col­lect responses to the mes­sage I sent to Media Lab com­mit­tee mem­bers with the sub­ject line shown above. It will not really make sense to non-sub­scribers, and: that’s okay!

One thread of my own first, not about query lan­guages exactly, but the data that feeds them:

Years ago, I read A Vast Machine by Paul N. Edwards, which has the sub­ti­tle “Computer Models, Cli­mate Data, and the Pol­i­tics of Global Warming.” It’s an account of, among other things, the early days of climate modeling — which has its roots mainly in nuclear strategy. The book also makes the case, persuasively, that there is no cli­mate data “out there” to be logged and analyzed; instead it must be “con­structed” in a very real way — which isn’t intended as some rel­a­tivist jab, but a plain descrip­tion of the seri­ous work required to trans­form the read­ings of buoys and bal­loons into some­thing a sci­en­tist can ask ques­tions about, and with.

It’s ter­rific for any­one inter­ested in the cli­mate or “big data” or both. More than most books I’ve read, fiction and non-fic­tion alike, it’s stayed in my mind, very much front and center, in all the years since I first encountered it.

Now, on to the responses … 

Pots and ves­sels from Tom:

What would it be like to “throw a mould” for the result to “grow” into? You cast not only a spell, but a shape, a pot, a struc­ture, and over time, the result fills into it. (I went to a pot­tery yes­ter­day and bits of it filled me with ideas, or some­thing.) [ … ]

Or, spin­ning a web, with struc­ture and shape, to see what data it catches.

Tom’s was the first reply I received, and it made me instantly glad I sent out this ques­tion!

Moths from Christine:

The “audio moth” wasn’t much like any­thing I was pic­tur­ing when I clicked through on your link, but those two words together spark a won­der­ful image for me. What if the data­base is queried/accessed through actual moths — or something alive, anyway, or alive-ish, where you might have to hold up a fin­ger for it to land on, buzzing and trem­bling as you speak your ques­tion, and then it floats off to con­sult the hive mind and return later with an answer. Some­thing ani­malistic, some­thing at least semi-sentient … how’s that for an inter­face?!

I like this vision because it reminds me of the queries I ran at Twit­ter ten years ago, sift­ing through the data­base of all tweets, and the hours I’d have to wait for them to complete. Most mod­ern com­put­ing has left that sense of “walking away from the computer” behind, for rea­sons of effi­ciency and interactivity, but of course, it has an aes­thetic appeal … 

Live streams from James:

One interesting strand of what you are describ­ing could be that a query is a living, dynamic thing. Per­haps every­thing this system observes is part of a flow of infor­ma­tion, and both the sys­tem and the query lan­guage are designed to orga­nize and visu­al­ize flows, rather than dis­crete values. A query might display the winds blow­ing across a continent, and as the winds change, so does the display. There would be operators to rewind the dis­play, but no con­cept of stor­ing or retriev­ing specific val­ues at a time, because why would that even be inter­est­ing? There would be oper­a­tors to look at changes over time, in which spe­cific incidents — a fire! a new kind of ani­mal — would stand out by their contrast. But there would be noth­ing like an arbi­trary SELECT … WHERE that pulls data out of their nat­ural context.

Similar sen­ti­ments from Maier, expressed a dif­fer­ent way:

The data is all exhaust of processes, which are only indi­rectly monitored. So I would want my query lan­guage to give me the processes. Think of a [Conway’s] Game of Life board. You have dead objects, you have sim­ple repet­i­tive patterns, like blinkers, and you also have more com­plex patterns, like gliders, and even glider guns. I would want to clas­sify these patterns [ … ] As an exam­ple query: please show me pat­terns of type X at scale Y with effect Z on area W. The next level up is ask­ing about inter­ac­tions between pat­terns.

The results should be shown as objects, not as pat­terns, and the rela­tion­ship of these objects to other items. An object is a pat­tern gen­er­a­tor which remains rather sta­ble over a query-relevant time scale.

Magical cor­re­spon­dence from Ben:

Something this makes me think of is that magic (fan­tasy) often has some­thing that pro­gram­ming (to a layperson) doesn’t … bear with me as I struggle to artic­u­late it … cor­re­spon­dence? Like sym­bol­ism but with concrete effects? Like if you’re going to cast a spell to move a mountain, you maybe need a stone from that mountain, or at least a stone — not just the mountain’s name or its coor­di­nates or URL or whatever. Usu­ally in cod­ing you just put together what­ever you need out of totally generic text. Which feels much less magical. Are there query lan­guages where to get data out you need to put data in that resembles, or is con­nected to, the data you want, in some way? Or used to be part of it? Shape your query with an example, ide­ally a con­crete exam­ple. Blockchain maybe has this resonance. This feels like very @rsnous territory.

I love this.

Webs from Steve:

You’ve likely seen arti­cles about spi­ders appar­ently use their webs as mem­ory and brain extensions? They receive input through the web vibrat­ing due to air move­ment or insects [ … ] your remark “maybe to make a query, you sing a song!” sug­gested “playing” a web as an instrument, or for communication.

Re: spiders, you might or might not be familiar with Eleanor Morgan’s Gossamer Days.

Maybe a query “shakes” the search area, and the replies sift or fall loose [ … ] from the dataset.

Bonsai from Nick:

With the sketch of this world you’ve offered, I wonder if an appro­pri­ate fic­ti­tious pro­gram­ming lan­guage may feature:

I love this, too, in part because I actu­ally do know some­thing about prun­ing trees, so I feel like I could bring real expe­ri­ence and tac­tile mem­ory to the table, imag­in­ing and describ­ing such a sys­tem. Always useful.

MRIs from Donna:

What I immediately thought of after read­ing your descrip­tion of your network was MRI diffusion ten­sor imaging — really beau­ti­ful images of the brain’s white matter. This article can explain bet­ter than I can, but I feel like this also adds onto Nick’s obser­va­tion with the bon­sai tree, where the infor­ma­tion builds itself up with­out explicit human direc­tion. For the brain, there are gen­eral motor and somatosen­sory areas and so on; each item in your nat­ural net­work can cor­re­spond to a dif­fer­ent zone. So as your data col­lects, occur­rences that hap­pen more often together would branch together and cre­ate lay­ers rein­forc­ing the connection. Heb­bian the­ory: neu­rons that fire together wire together.

Deletions from Rupert:

Back when I worked as an astronomer, the focus wasn’t on query­ing data per se, but delet­ing it. Much like (I imag­ine) your nat­ural plan­e­tary net­work, a tele­scope con­sumes such a colos­sal scale of data that it’s not fea­si­ble to deal with all of it. And importantly, there is always interference. I would use a cus­tom com­mand line tool to delete chunks of fre­quency data that were known inter­fer­ence bands, or which looked like they were phone calls or wifi sig­nals (you learn to recog­nise the char­ac­ter­is­tic spikes, and see the chunks hacked out of the wave­form where I removed them). Per­haps this ties into Nick’s com­ments on prun­ing — our future sci­en­tists aren’t select­ing from the data, but rather carv­ing it away to see what remains. A poten­tially risky dance of both inter­preting and cre­at­ing the data. A sculpture.

Analogies from Matt:

I don’t know any­thing about proteomics, or bio­log­i­cal data at all really; however, I’ve often made the argument — and it’s present in this manuscript, in a way — that the “real” appli­ca­tion of com­put­ers, in the full sweep of human history, will be biology, with effects so dra­matic it makes every other use (communications, entertainment, etc.) seem like a sideshow.

Provocations from David:

David’s first idea reminds me of the way Omar Rizwan—that’s his sec­ond cita­tion here — is often talking, or dreaming, about data with “provenance”: a deep mem­ory of how it was made, where it came from, how it’s been trans­formed.

Imagine show­ing up in response to a query with some sense of, “whew, took a long time to get here … I’m tired … ”

Error mes­sages from K. G. H.:

I think if you include some error mes­sages in your story, when bad syn­tax has unex­pected results or some such, it will tickle tech read­ers like me.

Legacy tech from Matt:

Technologies have momen­tum of a sort; I’m typ­ing this in an alpha­bet that was invented (in part, and improved along the way) that was invented in the Bronze Age. My the­ory (which I just made up, and could be entirely wrong) is that tab­u­lar data is kind of the same, in that it’ll be around for a long time. To back that up: we’ve already seen a cou­ple gen­er­a­tions of other types of data stores come and go. Remem­ber Mongo and Riak? Fas­ci­nat­ing tech, espe­cially the dis­trib­uted nature of Riak, but they weren’t suc­cess­ful because they were very hard to reason about.

I’m not sure this is quite what Matt intended, but I’m now imag­in­ing an aside that reveals this fan­tas­ti­cal data­base of the far future is running, at its root level … on CSV files.

AI from Dan:

Look, GPT-3 and gen­er­a­tive image “AI” are already the infi­nitely expres­sive query lan­guage and response palette you’re look­ing for!

Dataset: millions, millions, mil­lions of words and images from everywhere. And more, all the time. [ … ]

Query: nat­ural lan­guage, sure, but also your query is any­thing you can draw. That draw­ing gets filled in / auto­com­pleted [ … ] but only as a func­tion of the train­ing dataset.

So you already have a sort of proto: “I’ll draw a pas­toral scene, now you styl­is­ti­cally trans­form it based on, idk, weather data.”

Resonance from Matthew:

I’m instantly reminded of the con­cept of sto­chas­tic resonance, and the idea that you can apply cer­tain kinds of “noise” to a chaotic and noisy signal to make the strands you are looking for “hum” or res­onate enough to iden­tify and high­light the sig­nal you are look­ing for.

What if the way you query your nat­ural sys­tem is to inject cer­tain kinds noise to draw out and amplify the data in ways that you need?

And what if it was actu­ally dif­fer­ent kinds of gen­res of music/audio that worked best. Black Nor­we­gian death metal. Medieval harp music? 808 drum machine samples? Moog chords or midi sequences?

I’ve read a tiny bit about sto­chas­tic resonance, and it’s worth learn­ing about if you’ve never heard of it — a very deep magic trick.

Soapstone seals from Gabe:

It’s lovely to imag­ine a query being care­fully carved into soapstone, look­ing some­thing like an Indus Valley Civil­i­sa­tion seal: a depic­tion of an ani­mal or god, haloed by a string of abstract symbols. I’m imag­in­ing a query lan­guage as writ­ten in Indus Val­ley-esque char­ac­ters as well: abstract enough to be clearly symbolic, but fre­quently evoca­tive of the natural and supernat­ural world: a tree, a bird, a very tall crea­ture with many legs … 

[P]erhaps the seals rep­re­sent atomic units of queries, which you use to stamp sealings out of clay which you then arrange com­po­si­tion­ally in a mean­ing­ful way; queries them­selves are com­posed of seal­ings [ … ] fired in a kiln and arranged just so, placed some­where where they won’t be dis­turbed until a piece of the net­work wan­ders by and exe­cutes the query … 

Projections from Ruth:

The resolution/projection thing is clearly going to be some­thing big to tackle. Lots of local­ized datasets are pub­lished or processed at local projections, and obvi­ously look­ing at global data needs you to con­sider projections. If you’re calculating, like, % of trees on Earth by country, you want an area-accurate projection for the math but then your maps look like crap. How are you han­dling that? Is it built-in, or do you have to man­u­ally make those adjustments? This would be a really fun fan­tasy part! Could you just skip the whole pro­jec­tion thing by keep­ing the Earth a sphere and mak­ing all your math (and pixels) just accu­rately round? What a dream.

Regarding the sug­ges­tion in bold, I feel like I should just … do this! The com­pu­ta­tional sub­strate of this particular group, in this par­tic­u­lar future, isn’t dig­i­tal com­put­ers — and they don’t use “screens”—so, why not? THE DATA IS ROUND. I have decreed it.

Empathy from Duncan:

Finally, what would the out­put be like? I believe that a prob­lem with our cur­rent data par­a­digm is that it fails to generate, I’m not sure what to call it — empathy? Compassion? The envi­ron­ment is felt first and fore­most by the peo­ple it affects, and so any­one request­ing data should also have to feel the result of the query. If it’s a heatwave, you should feel the oppres­sive temperatures. If it’s a rainstorm, you should be drenched. If the air is polluted, [you should choke and wheeze].

A defense of SQL from Dmitry:

SQL may not be your favorite language, but it has a few attributes that I think are inter­est­ing for a fantastical language: in the­ory, at least, it is declarative — it describes the dataset one wants to exist, and, poof, if that dataset is satisfiable, it shows up. Now, in prac­tice, a SQL query pro­vides a lot of spe­cific details about where to fetch the data from, and how to relate and fil­ter dif­fer­ent datasets, and all that; and a fantastical lan­guage wouldn’t do any of that — but the idea, at its core, seems to me both fan­tas­ti­cal and futuristic. [ … ]

I really love that: “the idea, at its core, seems to me both fan­tas­ti­cal and futuristic”. Per­haps I ought to revise my esti­ma­tion of SQL upward, just a touch.

More from Dmitriy:

So, to finally get to my point: perhaps a spatial, envi­ron­men­tal, eco­log­i­cal query lan­guage for inter­act­ing with a mas­sive data­base, which is also a net­work and per­haps lends itself nat­urally to all sorts of gen­er­a­tive magic, is a descrip­tion of an envi­ron­ment that would con­tain the answer to the ques­tion you have in mind.

Fungal com­put­ing from Linda:

I immediately think of Andrew Adamatzky’s Towards fungal computer, where he pro­poses mush­rooms abil­ity to conduct elec­tric­ity could be used as a type of semi-con­ductor. So the hard­ware could be mycellium: “Likely appli­ca­tion domains of fungal devices could be large-scale net­works of mycelium which col­lect and analyse infor­ma­tion about environment of soil and, possibly, air and exe­cute some decision-mak­ing procedures. Fungi ‘possess almost all the senses used by humans’: Fungi sense light, chemicals, gases, grav­ity and elec­tric fields. Fungi show a pro­nounced response to changes in a sub­strate pH, demon­strate mechanosensing; they sense toxic metals, CO2 and direc­tion of fluid flow. [ … ] Fungi are also capa­ble of sens­ing chem­i­cal cues, espe­cially stress hormones, from other species, thus they might be used as reporters of health and well-being of other inhab­i­tants of the forest. Thus, fun­gal com­put­ers can be made an essen­tial part of dis­trib­uted large-scale envi­ron­mental sen­sor net­works in eco­log­i­cal research to assess not just soil qual­ity but the over­all health of the ecosys­tems.”

My “nat­ural net­work”, as presently described, is rhi­zomatic but not fun­gal, and I wonder if I ought to bring some fungi into the mix. (I hope you won’t think I’m a total fungi noob, though; many of these themes, these appreciations, are woven into Sourdough.)

Along the same lines, Diana sent along slime mold pics.

A Klin­gon pro­gram­ming lan­u­age (!) from Billy:

I used to think about this kind of thing a load back in my uni days in the late 90’s when I was study­ing com­put­ing, and was very pleased when I found var’aq a few years later — a Klingon pro­gram­ming language, try­ing to fit in with the cul­ture and lan­guage, and feel as if it had spring from the mind of a fic­tional Klin­gon Den­nis Ritchie. I’ve never found any­one else who’s heard of it, so it might have slipped under your radar. Unfortunately, it didn’t really go anywhere, and while look­ing for more about it just now, I found not much more than the old Geoc­i­ties pages where it used to live, which I still remember.

What a treat. I recommend pok­ing around the var’aq page if you’re even a lit­tle bit curi­ous; it’s a gen­uinely nice piece of work.

Layers from an anonymous cor­re­spondent:

In my job, I’m reg­u­larly asked to spew out dump of flat­tened data that I grab via SQL and some­one else ends up mas­sag­ing via pivot tables and arcane [business intelligence] tools I’m only vaguely famil­iar with. That’s mostly to say that a REAL feel­ing query inter­face would prob­a­bly have similar lay­ers.

I take that as a nudge to think about the entire “social reality” of a sys­tem of this kind, not just the tech­ni­cal spe­cifications of one layer. I, too, have been a sin­gle human step in an unruly data cascade; there are nar­ra­tive oppor­tu­ni­ties there, drama and humor … 

I received many links to real exist­ing pro­gram­ming lan­guages worth look­ing at:

Via Alisdair, SHRDLU. (A demo here—what a vibe.)

Via Andy, FORTH.

Via Greg, Prolog:

It is a kind of query lan­guage, where you enter a bunch of facts and rules and then ask if something is true. It always seemed like a really cool idea, but in prac­tice it was slow and when some­thing went wrong it was very dif­fi­cult to tell why!

Via Jared, Datalog:

I’ve been really inter­ested in a query lan­guage called Datalog, that works via a “pat­tern-matching” type approach. So you have “holes” in your query, which are “filled” with the data that match them in your data­base.

So I’m imag­in­ing in this fan­tasy world you have to con­struct a pattern ecosystem in order to query the nat­ural data­base. Like a little ecosys­tem enclosed in a sphere, with some key pieces replaced with some totem that rep­re­sents a variable, or “hole” in the pattern. So more com­plex queries would require many of these little terrariums, each shar­ing linked totems. You could imple­ment grow­ing or evolv­ing queries as just leav­ing these lit­tle pat­tern ecosys­tems alone for a while.

Maybe going too deep, but I can sorta see a fuzzy ver­sion of this that works in a web browser where you make a series of lit­tle ecosys­tems, maybe on a tile grid, which it then com­piles into queries and exe­cutes. [ … ]

Via Tom, the eso­teric pro­gram­ming lan­guages wiki, whoa!

That’s it for now!

Sent to the Media Lab committee in September 2021