The Media Lab
Fantasy query languages
This is where I’ll collect responses to the message I sent to Media Lab committee members with the subject line shown above. It will not really make sense to non-subscribers, and: that’s okay!
One thread of my own first, not about query languages exactly, but the data that feeds them:
Years ago, I read A Vast Machine by Paul N. Edwards, which has the subtitle “Computer Models, Climate Data, and the Politics of Global Warming.” It’s an account of, among other things, the early days of climate modeling —
It’s terrific for anyone interested in the climate or “big data” or both. More than most books I’ve read, fiction and non-fiction alike, it’s stayed in my mind, very much front and center, in all the years since I first encountered it.
Now, on to the responses …
Pots and vessels from Tom:
What would it be like to “throw a mould” for the result to “grow” into? You cast not only a spell, but a shape, a pot, a structure, and over time, the result fills into it. (I went to a pottery yesterday and bits of it filled me with ideas, or something.) [ … ]
Or, spinning a web, with structure and shape, to see what data it catches.
Tom’s was the first reply I received, and it made me instantly glad I sent out this question!
Moths from Christine:
The “audio moth” wasn’t much like anything I was picturing when I clicked through on your link, but those two words together spark a wonderful image for me. What if the database is queried/accessed through actual moths —
or something alive, anyway, or alive-ish, where you might have to hold up a finger for it to land on, buzzing and trembling as you speak your question, and then it floats off to consult the hive mind and return later with an answer. Something animalistic, something at least semi-sentient … how’s that for an interface?!
I like this vision because it reminds me of the queries I ran at Twitter ten years ago, sifting through the database of all tweets, and the hours I’d have to wait for them to complete. Most modern computing has left that sense of “walking away from the computer” behind, for reasons of efficiency and interactivity, but of course, it has an aesthetic appeal …
Live streams from James:
One interesting strand of what you are describing could be that a query is a living, dynamic thing. Perhaps everything this system observes is part of a flow of information, and both the system and the query language are designed to organize and visualize flows, rather than discrete values. A query might display the winds blowing across a continent, and as the winds change, so does the display. There would be operators to rewind the display, but no concept of storing or retrieving specific values at a time, because why would that even be interesting? There would be operators to look at changes over time, in which specific incidents —
a fire! a new kind of animal — would stand out by their contrast. But there would be nothing like an arbitrary SELECT … WHERE that pulls data out of their natural context.
Similar sentiments from Maier, expressed a different way:
The data is all exhaust of processes, which are only indirectly monitored. So I would want my query language to give me the processes. Think of a [Conway’s] Game of Life board. You have dead objects, you have simple repetitive patterns, like blinkers, and you also have more complex patterns, like gliders, and even glider guns. I would want to classify these patterns [ … ] As an example query: please show me patterns of type X at scale Y with effect Z on area W. The next level up is asking about interactions between patterns.
The results should be shown as objects, not as patterns, and the relationship of these objects to other items. An object is a pattern generator which remains rather stable over a query-relevant time scale.
Magical correspondence from Ben:
Something this makes me think of is that magic (fantasy) often has something that programming (to a layperson) doesn’t … bear with me as I struggle to articulate it … correspondence? Like symbolism but with concrete effects? Like if you’re going to cast a spell to move a mountain, you maybe need a stone from that mountain, or at least a stone —
not just the mountain’s name or its coordinates or URL or whatever. Usually in coding you just put together whatever you need out of totally generic text. Which feels much less magical. Are there query languages where to get data out you need to put data in that resembles, or is connected to, the data you want, in some way? Or used to be part of it? Shape your query with an example, ideally a concrete example. Blockchain maybe has this resonance. This feels like very @rsnous territory.
I love this.
Webs from Steve:
You’ve likely seen articles about spiders apparently use their webs as memory and brain extensions? They receive input through the web vibrating due to air movement or insects [ … ] your remark “maybe to make a query, you sing a song!” suggested “playing” a web as an instrument, or for communication.
Re: spiders, you might or might not be familiar with Eleanor Morgan’s Gossamer Days.
Maybe a query “shakes” the search area, and the replies sift or fall loose [ … ] from the dataset.
Bonsai from Nick:
With the sketch of this world you’ve offered, I wonder if an appropriate fictitious programming language may feature:
the OS offering up suggested queries/observations based on patterns in the data it’s seen on its own. The human is there to interpret and prune rather than enter queries; the OS does the heavy hypothesis-building.
the OS manifests these patterns as physical growths, like a bonsai [tree]. The trunk represents a high-confidence finding, each branch a curious consequence, each twig an unlikely but beautiful possibility. The human then prunes/cuts/trains the plant, and coaxes it in the direction of truth.
I love this, too, in part because I actually do know something about pruning trees, so I feel like I could bring real experience and tactile memory to the table, imagining and describing such a system. Always useful.
MRIs from Donna:
What I immediately thought of after reading your description of your network was MRI diffusion tensor imaging —
really beautiful images of the brain’s white matter. This article can explain better than I can, but I feel like this also adds onto Nick’s observation with the bonsai tree, where the information builds itself up without explicit human direction. For the brain, there are general motor and somatosensory areas and so on; each item in your natural network can correspond to a different zone. So as your data collects, occurrences that happen more often together would branch together and create layers reinforcing the connection. Hebbian theory: neurons that fire together wire together.
Deletions from Rupert:
Back when I worked as an astronomer, the focus wasn’t on querying data per se, but deleting it. Much like (I imagine) your natural planetary network, a telescope consumes such a colossal scale of data that it’s not feasible to deal with all of it. And importantly, there is always interference. I would use a custom command line tool to delete chunks of frequency data that were known interference bands, or which looked like they were phone calls or wifi signals (you learn to recognise the characteristic spikes, and see the chunks hacked out of the waveform where I removed them). Perhaps this ties into Nick’s comments on pruning —
our future scientists aren’t selecting from the data, but rather carving it away to see what remains. A potentially risky dance of both interpreting and creating the data. A sculpture.
Analogies from Matt:
The problem of querying within a massive dataset, so diffuse and noisy that any conclusions can be at best statistical rather than certain, suggests that you may find some inspiration in the field of proteomics. Here’s a potential starting point.
What if the fantasy queries were topological rather than linguistic? What if they described a particular shape in the data and locked to it, like an antibody to a spike protein?
What if that shape were described as an intricate n-dimensional mesh?
I don’t know anything about proteomics, or biological data at all really; however, I’ve often made the argument —
Provocations from David:
Suppose the data from your [sensors] is not just about the actual data, but more about how it traverses the network to arrive —
sort of a LoRa thing.
Suppose the query language is more along the line of a proof —
instead of receiving a dataset in response to your query, you get a true/false proof of your hypothesis.
David’s first idea reminds me of the way Omar Rizwan—that’s his second citation here —
Imagine showing up in response to a query with some sense of, “whew, took a long time to get here … I’m tired … ”
Error messages from K. G. H.:
I think if you include some error messages in your story, when bad syntax has unexpected results or some such, it will tickle tech readers like me.
Legacy tech from Matt:
Technologies have momentum of a sort; I’m typing this in an alphabet that was invented (in part, and improved along the way) that was invented in the Bronze Age. My theory (which I just made up, and could be entirely wrong) is that tabular data is kind of the same, in that it’ll be around for a long time. To back that up: we’ve already seen a couple generations of other types of data stores come and go. Remember Mongo and Riak? Fascinating tech, especially the distributed nature of Riak, but they weren’t successful because they were very hard to reason about.
I’m not sure this is quite what Matt intended, but I’m now imagining an aside that reveals this fantastical database of the far future is running, at its root level … on CSV files.
AI from Dan:
Look, GPT-3 and generative image “AI” are already the infinitely expressive query language and response palette you’re looking for!
Dataset: millions, millions, millions of words and images from everywhere. And more, all the time. [ … ]
Query: natural language, sure, but also your query is anything you can draw. That drawing gets filled in / autocompleted [ … ] but only as a function of the training dataset.
So you already have a sort of proto: “I’ll draw a pastoral scene, now you stylistically transform it based on, idk, weather data.”
Resonance from Matthew:
I’m instantly reminded of the concept of stochastic resonance, and the idea that you can apply certain kinds of “noise” to a chaotic and noisy signal to make the strands you are looking for “hum” or resonate enough to identify and highlight the signal you are looking for.
What if the way you query your natural system is to inject certain kinds noise to draw out and amplify the data in ways that you need?
And what if it was actually different kinds of genres of music/audio that worked best. Black Norwegian death metal. Medieval harp music? 808 drum machine samples? Moog chords or midi sequences?
I’ve read a tiny bit about stochastic resonance, and it’s worth learning about if you’ve never heard of it —
Soapstone seals from Gabe:
It’s lovely to imagine a query being carefully carved into soapstone, looking something like an Indus Valley Civilisation seal: a depiction of an animal or god, haloed by a string of abstract symbols. I’m imagining a query language as written in Indus Valley-esque characters as well: abstract enough to be clearly symbolic, but frequently evocative of the natural and supernatural world: a tree, a bird, a very tall creature with many legs …
[P]erhaps the seals represent atomic units of queries, which you use to stamp sealings out of clay which you then arrange compositionally in a meaningful way; queries themselves are composed of sealings [ … ] fired in a kiln and arranged just so, placed somewhere where they won’t be disturbed until a piece of the network wanders by and executes the query …
Projections from Ruth:
The resolution/projection thing is clearly going to be something big to tackle. Lots of localized datasets are published or processed at local projections, and obviously looking at global data needs you to consider projections. If you’re calculating, like, % of trees on Earth by country, you want an area-accurate projection for the math but then your maps look like crap. How are you handling that? Is it built-in, or do you have to manually make those adjustments? This would be a really fun fantasy part! Could you just skip the whole projection thing by keeping the Earth a sphere and making all your math (and pixels) just accurately round? What a dream.
Regarding the suggestion in bold, I feel like I should just … do this! The computational substrate of this particular group, in this particular future, isn’t digital computers —
Empathy from Duncan:
Finally, what would the output be like? I believe that a problem with our current data paradigm is that it fails to generate, I’m not sure what to call it —
empathy? Compassion? The environment is felt first and foremost by the people it affects, and so anyone requesting data should also have to feel the result of the query. If it’s a heatwave, you should feel the oppressive temperatures. If it’s a rainstorm, you should be drenched. If the air is polluted, [you should choke and wheeze].
A defense of SQL from Dmitry:
SQL may not be your favorite language, but it has a few attributes that I think are interesting for a fantastical language: in theory, at least, it is declarative —
it describes the dataset one wants to exist, and, poof, if that dataset is satisfiable, it shows up. Now, in practice, a SQL query provides a lot of specific details about where to fetch the data from, and how to relate and filter different datasets, and all that; and a fantastical language wouldn’t do any of that — but the idea, at its core, seems to me both fantastical and futuristic. [ … ]
I really love that: “the idea, at its core, seems to me both fantastical and futuristic”. Perhaps I ought to revise my estimation of SQL upward, just a touch.
More from Dmitriy:
So, to finally get to my point: perhaps a spatial, environmental, ecological query language for interacting with a massive database, which is also a network and perhaps lends itself naturally to all sorts of generative magic, is a description of an environment that would contain the answer to the question you have in mind.
Fungal computing from Linda:
I immediately think of Andrew Adamatzky’s Towards fungal computer, where he proposes mushrooms ability to conduct electricity could be used as a type of semi-conductor. So the hardware could be mycellium: “Likely application domains of fungal devices could be large-scale networks of mycelium which collect and analyse information about environment of soil and, possibly, air and execute some decision-making procedures. Fungi ‘possess almost all the senses used by humans’: Fungi sense light, chemicals, gases, gravity and electric fields. Fungi show a pronounced response to changes in a substrate pH, demonstrate mechanosensing; they sense toxic metals, CO2 and direction of fluid flow. [ … ] Fungi are also capable of sensing chemical cues, especially stress hormones, from other species, thus they might be used as reporters of health and well-being of other inhabitants of the forest. Thus, fungal computers can be made an essential part of distributed large-scale environmental sensor networks in ecological research to assess not just soil quality but the overall health of the ecosystems.”
My “natural network”, as presently described, is rhizomatic but not fungal, and I wonder if I ought to bring some fungi into the mix. (I hope you won’t think I’m a total fungi noob, though; many of these themes, these appreciations, are woven into Sourdough.)
Along the same lines, Diana sent along slime mold pics.
A Klingon programming lanuage (!) from Billy:
I used to think about this kind of thing a load back in my uni days in the late 90’s when I was studying computing, and was very pleased when I found var’aq a few years later —
a Klingon programming language, trying to fit in with the culture and language, and feel as if it had spring from the mind of a fictional Klingon Dennis Ritchie. I’ve never found anyone else who’s heard of it, so it might have slipped under your radar. Unfortunately, it didn’t really go anywhere, and while looking for more about it just now, I found not much more than the old Geocities pages where it used to live, which I still remember.
What a treat. I recommend poking around the var’aq page if you’re even a little bit curious; it’s a genuinely nice piece of work.
Layers from an anonymous correspondent:
In my job, I’m regularly asked to spew out dump of flattened data that I grab via SQL and someone else ends up massaging via pivot tables and arcane [business intelligence] tools I’m only vaguely familiar with. That’s mostly to say that a REAL feeling query interface would probably have similar layers.
I take that as a nudge to think about the entire “social reality” of a system of this kind, not just the technical specifications of one layer. I, too, have been a single human step in an unruly data cascade; there are narrative opportunities there, drama and humor …
I received many links to real existing programming languages worth looking at:
Via Alisdair, SHRDLU. (A demo here—what a vibe.)
Via Andy, FORTH.
Via Greg, Prolog:
It is a kind of query language, where you enter a bunch of facts and rules and then ask if something is true. It always seemed like a really cool idea, but in practice it was slow and when something went wrong it was very difficult to tell why!
Via Jared, Datalog:
I’ve been really interested in a query language called Datalog, that works via a “pattern-matching” type approach. So you have “holes” in your query, which are “filled” with the data that match them in your database.
So I’m imagining in this fantasy world you have to construct a pattern ecosystem in order to query the natural database. Like a little ecosystem enclosed in a sphere, with some key pieces replaced with some totem that represents a variable, or “hole” in the pattern. So more complex queries would require many of these little terrariums, each sharing linked totems. You could implement growing or evolving queries as just leaving these little pattern ecosystems alone for a while.
Maybe going too deep, but I can sorta see a fuzzy version of this that works in a web browser where you make a series of little ecosystems, maybe on a tile grid, which it then compiles into queries and executes. [ … ]
Via Tom, the esoteric programming languages wiki, whoa!
That’s it for now!
Sent to the Media Lab committee in September 2021