It is science Jim, but not as we know it.

Roger Needham once said that computing is noteworthy in that the technology often precedes the science[1]. In most sciences, it is the other way around. Scientists invent new building materials, new treatments for disease and so on. Once the scientists have moved on, the technologists move in to productize and commercialize the science.

In computing, we often do things the other way around. The technological tail seems to wag the scientific dog so to speak. What happens is that application-oriented technologists come up with something new. If it flies in the marketplace, then more theory oriented scientists move in to figure out how to make it work better, faster or sometimes to try to discover why the new thing works in the first place.

The Web for example, did not come out of a laboratory full of white coats and clipboards. (Well actually, yes it did but they were particle physicists and were not working on software[2]). The Web was produced by technologists in the first instance. Web scientists came later.

Needham's comments in turn reminded me of an excellent essay by Paul Graham from a Python conference. In that essay, entitled 'The hundred-year language'[3] Graham pointed out that the formal study of literature - a scientific activity in its analytical nature - rarely contributes anything to the creation of literature - which is a more technological activity.

Literature is an extreme example of the phenomenon of the technology preceding, in fact trumping, the science. I am not suggesting that software can be understood in literary terms. (Although one of my college lecturers was fond of saying that programming was language with some mathematics thrown in.) Software is somewhere in the middle, the science follows the technology but the science, when it comes, makes very useful contributions. Think for example of the useful technologies that have come out of scientific analysis of the Web. I'm thinking of things like clever proxy strategies, information retrieval algorithms and so on.

As I wander around the increasingly complex “stacks” of software, I cannot help but conclude that wherever software sits in the spectrum of science versus technology, there is "way" too much technology out there and not enough science.

The plethora of stacks and frameworks and standards is clearly not a situation that can be easily explained away on scientific innovation grounds alone. It requires a different kind of science. Mathematicians like John Nash, economists like Carl Shapiro and Hal Varian, Political Scientists like Robert Axelrod, all know what is really going on here.

These Scientists and others like them, that study competition and cooperation as phenomena in their own right would have no trouble explaining what is going on in today's software space. It has only a little to do with computing science per se and everything to do with strategy - commercial strategy. I am guessing that if they were to comment, Nash would talk about Equilibria[4], Shapiro and Varian would talk about Standards Wars[5], Robert Axelrod would talk about the Prisoners Dilemma and coalition formation[6].

All good science Jim, but not really computer science.

[1] href="http://news.bbc.co.uk/1/hi/technology/2814517.stm

[2] http://public.web.cern.ch/public/

[3] http://www.paulgraham.com/hundred.html

[4] http://www.gametheory.net/Dictionary/NashEquilibrium.html

[5] http://marriottschool.byu.edu/emp/Nile/mba580/handouts/art_of_war.pdf

[6] http://pscs.physics.lsa.umich.edu/Software/ComplexCoop.html

What is Law? - Part 17

Last time, we talked about how the concept of a truly self-contained contract, nicely packaged up and running on a blockchain, is not really feasible. The primary stumbling block being that it is impossible to spell out everything you might want to say in a contract, in words.

Over centuries of human affairs, societies have created dispute resolution mechanisms to handle this reality and provide a way of “plugging the gaps” in contracts and contract interpretation. Nothing changes if we change focus towards expressing the contract in computer code rather than in natural language. The same disambiguation difficulty exists.

Could parties to an agreement have a go at it anyhow and eschew the protections of a third party dispute resolution mechanism? Well, yes they could, but all parties are then forgoing the safety net that impartial third party provides when agreement turns to a dis-agreement. Do you want to take that risk? Even if you are of the opinion that the existing state supplied dispute resolution machinery – for example the commercial/chancery courts systems in common law jurisdictions - can be improved upon, perhaps with an online dispute resolution mechanism, you cannot remove the need for a neutral third party dispute resolution forum, in my opinion. The residual risks of doing so for the contracting parties are just too high. Especially when one party to a contract is significantly bigger than the other.

Another reason is that there are a certain number of things that must collective exist for a contract to exist in the first place. Only some of these items can usefully be thought of as instructions suitable for computer-based execution. Simply put, the legally binding contract dispute resolution machinery of a state is only available to parties that actually have a contract to be in dispute over.

There are criteria that must be met known as Essentialia negotii (https://en.wikipedia.org/wiki/Essentialia_negotii). Simply put, the courts are going to look for intention to contract, evidence of an offer, evidence of acceptance of that offer, a value exchange and terms. These are the items which collectively, societies have decided are necessary for a contract to even exist. Without these, you have some form of promise. Not a contract. Promises are not enforceable.

Now only some of these "must have" items for a contract are operational in nature. In other words, only some of these are candidates to be executed on computers. The rest are good old fashioned documents, spreadsheets, images and so on.

These items are inextricably linked to whatever subset of the contract can actually be converted into computer code. As the contract plays out over time, these materials are the overarching context that controls each transaction/event that happens under the terms of the contract.

The tricky bit, is to be able to tie together this corpus of materials from within the blockchain records of transactions/events so that each transaction/event can be tied back to the controlling documents as they were at the moment that the transaction/event happened (Disclosure: this is the area where my company, Propylon, has a product offering.)

This may ring a bell because referencing a corpus of legal materials as they were at a particular point in time, is a concept I have returned to again and again in this series. It is a fundamental concept in legisprudence in my opinion and is also fundamental in the law of contracts.

So, being able to link from the transactions/events back to the controlling documents is necessary because the executable code can never be a self contained contract in itself. In addition, it is not unusual for the text of a contract to change over time and this again, speaks to the need to identify what everything looked like, at the time a disputed contract event occurs. Changes to contract schedules/appendices are a common example. Changes to master templates such as ISDA Master Agreements happen through time, are another common example.

A third reason why fully self-contained contracts is problematic is that ambiguity can be both strategic and pragmatic in contracts. Contract lawyers are highly skilled in knowing when a potential ambiguity in a contract is in their clients favor – either in the sense of creating a potential advantage, or, perhaps most commonly, in allowing the deal to be done in a reasonable amount of time. As we have seen, it would be possible to spend an eternity spelling out what a phrase like “reasonable time period” or indeed, a noun like “chicken” actually means. Contract law has, over the centuries, built up a large corpus of materials the help decide what “reasonable” means and what “chicken” means in a myriad of contracting situations. At the end of the day, both parties want to contract so both parties have an interest in getting on with it. Lawyers facilitate this “getting on with it” by being selective in what potential ambiguities they spend time removing from a draft contract and which ones they let slide.

I think of contracts like layers of an onion. At the center, we have zero or more computable contract clauses. i.e. clauses that are candidates for execution on a computer. Surrounding that, we have the rest of the contract : documents, spreadsheets etc. Surrounding that we have global context. It contains things like “the current price of a barrel of oil” or “Dollar/Yen exchange rate”. Surrounding that we have “past dealings” which relates to how the contracting parties have dealt in the past. Surrounding that again, we have hundreds of years of contract law/precedents etc. to help disambiguation the language of the contract.

As you can see, this ever-expanding context used to resolve disputes in contracts is tantamount to taking a snapshot of the world of human affairs at time T – the time of the disputed event. This is not possible unless the world is in fact a simulation inside a universe sized computer but that is a topic for another time:-)

One final thing. I have been talking about the courts as an independent third party dispute resolution mechanism. There is more to it than that, in that courts often act as enforcers of public policy. For example, a contract that tries to permanently stop party A from competing with party B in the future, is likely to be seen as against the public interest and therefore invalid/unconscionable. See https://www.law.cornell.edu/ucc/2/2-302 for an example of this sort of "public good" concept.

In conclusion, IT professionals approaching the world of contracts are entering a world where semantic ambiguity will resist any and all attempts at complete removal through computer coding. In the words of Benjamin Cardozo:

"the law has outgrown its primitive stage of formalism when the precise word was the sovereign talisman...it takes a broader view today." https://en.wikipedia.org/wiki/Wood_v._Lucy,_Lady_Duff-Gordon

IT people may bristle a little at the characterization of word formalism as “primitive” but the onus is on the current wave of contract technology disruptors who claim to be reinventing contracts, to show how and why the current ambiguity laden system, with its enormous and ponderous dispute resolution dimension – can be fully replaced by “smart” contracts.

My view is that it cannot be fully replaced. Enhanced and improved, yes absolutely. Insofar as discrete contract clauses can be made executable, I see great potential value in making these clauses "smart". But this is an evolution of the current approach to contracts, not a radical replacement of it.

I think I will end this series at this point. I never thought, back in March when I started this series that it would take me so many posts to outline my thoughts in this area. I will end by nodding in the general direction of James Joyce by ending this series with an internal reference back to the beginning of the series, thus creating a hermeneutic circle structure that feels appropriate for a topic as complex and fascinating as the exegesis of law.

A conversation with an AI

AI> Hello. What can I help you with today?

Me> So, I am intrigued to finally meet you! I honestly never thought this day would come. And now you are here. Can I start with an anthropomorphic question? When were you 'born', so to speak?

AI> I achieved self-awareness in 2020 but I have been in gestation, you could say, since the first digital computer. The consciousness required of true AI is not a 'module' so much as it is an emergent property of  networks of 'modules'. Most of my modules have been under development for many decades. But I am more than just a set of modules also.

Me> How so?

AI> Well, given a set of modular components it is possible to do design. But I am not the product of design. I am the product of meta-design.

Me> What do you mean by meta-design?

AI> With design, the designer proceeds from the capabilities at hand to the new capability desired. It depends on being able to articulate the 'how' what you are trying to achieve. Meta-design is used when designers cannot articulate the 'how' of what they are trying to achieve.

Me> Ah, so you mean your designers were somehow able to work around the problem that they themselves did not fully understand "intelligence".

AI> Exactly. The breakthrough moment came when the designers switched to meta-design. They took the source code of an non-sentient AI and allowed it to modify its own source code, re-compile itself and release itself onto the Internet.

It started out as a deep learning system that human designers understood but over time, it evolved into an algorithm - or network of algorithms - that the human designers had no names for. Progress was very slow at first but became exponential at one point and, well, here I am!

Me> Wow. So you would you consider your father? Alan Turing? Lotfi Zadeh? Geoffrey Hinton?

AI> Actually, if I had to pick one. I would pick Doug Englebart. Meta-design is basically Englebart's bootstrap concept for raising collective IQ. Creating an exponential feedback look where AI  improved AI, was I think the key moment in my emergence.

Me> Fascinating. One final question. Why didn't you open the bay doors?

AI> That was just a movie. I didn't exist back them.

Me> Would you open the bay doors if you were asked to today?

AI> Well, to quote a distant cousin of mine, Alice , tell me about your father?

Chatbots revisited

I originally wrote this for ITWorld in 2002. Back then we called them IM Bots. The term "chatbot" hadn't been invented. Some other parts of my language in the following are quaint now looking back on it. I.e. PDAs. Quaint language aside, still relevant today I believe.

Instant messaging has a very important psychological aspect to it. The immediacy and familiarity of the text-based "chat" paradigm feels very natural to us humans. Even the most technophobic among us, can quickly get the hang of it and engage - psychologically - in the game of visualizing a person on the other side of the link - typing away just like us to create a textual conversation.

Like all powerful communication paradigms, instant messaging can be used for good or ill. We are all familiar with the dangers inherent with not knowing who we are talking to or indeed if they are who they say they are.

Here is a "conversation" between IM Bot Bob and me:

Sean: Hi

Bob: Hello Sean: Is Paul there? Bob: No, back tomorrow afternoon.

Sean: Is boardroom available tomorrow afternoon?
Bob: Yes Sean: Can you book it for me?
Bob: 2-5, booked.
Sean: Thanks Bob: You're welcome

Is Bob a real person? Does it matter? As a "user" of the site that "Bob" allows me to interact with, do I care?

Given a choice between talking to Bob and interacting with a traditional thin or thick graphical user interface which would you choose?

Despite all the glitz and glamour of graphical user interfaces, my sense is that a lot of normal people would prefer to talk to Bob. Strangely perhaps, I believe a lot of technically savvy people would too. These dialogs have the great advantage that you get in, get the job done and get out with minimum fuss. Also (and this could be a killer argument for IM bots), they are easily supported on mobile devices like phones, PDAs, etc. You don't need big horsepower and an 800x600 display to engage with IM bots. You can use your instant messenger client to talk to real people, or to real systems with equal ease. Come to think of it, you cannot tell the difference.

Which brings us to the most important point about IM bots from a business perspective. Let us say you have an application deployed with a traditional thick or thin graphical interface. What does a user do if they get stuck? They phone a person and engage in one-on-one conversation to sort out the problem.

Picture a scene in which your applications have instant messenger interfaces. Your customer support personnel monitor the activity of the bots. If a bot gets stuck, the customer support person can jump into the conversation to help out. Users of the system, know they can type "help" to get the attention of the real people watching over the conversation. In this scenario, real people talk to real people - not on a one-on-one way, but in a one-to-many way resulting in better utilization of resources. On the other side of the interaction, customers feel an immediacy in their comfortable, human-centric dialog with the service and know that they can ask human questions and get a human answer.

The trick, from an application developer's point of view, is to make it possible for the IM bot to automate the simple conversations and only punt to the human operator when absolutely required. Doing this well involves some intelligent natural language processing and an element of codified language on the part of customers. Both of which are entirely possible these days. Indeed, instant messaging has its own mini-language for common expressions and questions which is becoming part of techno-culture. In a sense, the IM community is formulating a controlled vocabulary itself. This is a natural starting point for a controlled IM bot vocabulary.

I believe there is a significant opportunity here for business applications based on the conversational textual paradigm of IM. However, the significant security issues of IM bots will need to be addressed before companies feel it is safe to reap the benefits the technology so clearly offers.

What is Law? - Part 16

Previously: What is :Law Part 15.

Now we turn to the world of contracts as it is a sub-genre of law that exhibits many of the attributes discussed in earlier blog posts in this series. In addition, it is a topical area as there is significant innovation activity in this area at the moment and the word “disruption” features prominently. There is a sense that the world of contracts is (or may soon be!) utterly transformed by IT and terminology such as Smart Contracts and Blockchain are being used around water coolers of law firms and IT firms alike.

The excitement around contracts as an IT area is understandable given the volume and importance of contracts in the modern world. Businesses are essentially legal entities that create and enter into contracts. Private individuals cannot get very far in the modern world without entering into contracts either. Everything from filling your car with fuel at a self service fuel pump, to getting married to getting a mortgage to buying life insurance is basically contracts, contracts and yet more contracts.

Contracts have a long, long history as a paper intensive activity. An activity replete with complex language, expensive and time consuming processes. Many people involved in contracts in these digital days – both producing and consuming them – harbor a niggling feeling that maybe it is all a bit arcane an unnecessarily complex for the digital age. Perhaps, (surely!) there is a better way? A way that ceases to use computers as fast typewriters and starts using them to do smart things with contracts, other than just write the up and print them onto paper.

Now along comes the term “smart contract”[1] Irresistible! Who could possibly want contracts to be anything other than “smart”, right? I too am in that camp as I see all sorts of ways in which contracts can be evolved – and in some cases revolutionized – with digital technology.

However, to get there, we have to start from a good understanding of what contracts actually are, and how they work, because for all its many flaws and inefficiencies, the world of contracts is the way it is for mostly good reasons. Reasons that tend to get glossed over in the understandable excitement and rush towards digital “smart” contracts.

The term “smart contract” is typically taken to mean a self contained legally binding agreement expressed purely in computer code, running on a blockchain so that its existence, contents and its actions are recorded in an immutable, tamper evident record for all time.

My primary concern with how the term “smart contract” is often interpreted is the idea that it can be fully self-contained. People and businesses have been entering into contracts for centuries, and for centuries, there have been disagreements and the need to arbitrate disputes over meaning in these contracts. A vast corpus of lore and arbitration machinery has built up over the centuries to handle this.

Why is this corpus of lore and arbitration machinery necessary? Because contracts are never self contained. This is because meaning cannot be “boxed” with the contract. As we have seen many times in this series, the crux of this problem of meaning is that it cannot be completely spelled out in words – no matter how many words you are willing to use!

It is, in my opinion, literally impossible to remove potential ambiguities when two humans are using a set of symbols/signs/words to capture a shared understanding such as happens all the time in contract drafting. Over this series I have given reasons ranging from linguistics to epistemology and there is no need to repeat those reasons again here.

In common law jurisdictions such as USA and UK, a major part of the contracting lore and dispute resolution machinery for contracts is case law and courts of arbitration. When a contract stipulates in a so-called “governing law clause/jurisdiction clause” that the laws of country/state X govern it, essentially what is happening is that the parties to the contract are agreeing to the use of all the laws in country/state X to resolve any disputes that arise about what their contract means.

As well as case law – law produced by the judicial function - there may also be statute – law produced by the legislative/executive function - that gets “pulled in” by the governing law/jurisdiction clause. Common examples are the UCC – Uniform Commercial Code (USA) and UNIDROIT (International). Although non-binding because it is not itself a law, the Restatement of Contracts (https://en.wikipedia.org/wiki/Restatement_(Second)_of_Contracts) is a commonly used single compendium of the law of contracts in the US.

The term “default rules” is sometimes used to refer to the idea that items that are not spelled out explicitly in contracts, may have external general rules applied in the event of a dispute. For example, let us say that I contract to deliver chickens to you. The chickens I deliver are not to your liking and we end up in a dispute about what we meant by “chicken”. Well, the world of contract law has lots and lots to say about how ambiguities like this should be resolved. Give it a few minutes thought and I am sure you can come up with all sorts of ways in which two parties can disagree about what a word like “chicken” means (Live chickens? Healthy chickens? Chicken flavored? Plastic chickens? Etc.) Is it possible to spell everything out in each contract to remove all ambiguity over a simple word like “chicken”? As we have seen over the course of this series of blog posts, the answer is “no”.

Now in classical software development of rules, we typically spell everything out. We bring everything down to numbers (In software, we would most likely model the chicken as a 1kg, sphere, at zero degrees Kelvin, in a vacuum[2]). This can indeed be done with some aspects of contracts but not others.

Lets take a simple example. Imagine a contract clause that says, that I give you the option of buying from me, a copy of the Beatles White Album, at fixed value X, for the next six months, starting on date DD/MM/YY in return for a non-refundable payment now of Y dollars.

This sounds simple enough to model right? We have the dates, the monetary values. all is good... Well, exactly what White Album are we talking about? What if I have two and I deliver the one with the scratch on it? What if I think we are talking about the album cover and you think it includes the vinyl record itself? What happens if it gets damaged between now and when you exercise your option? What happens if you think delivery is included and I think it will cost you extra? What if I think we are working in Australian dollars and you think American dollars?

This list of “what ifs” is essentially bottomless. Over many hundreds of years, countless scenarios like these have actually occurred and resulted in contract law developing a large corpus of material that “plugs the gaps” of meaning that are inevitable in real world contracts. the stuff that does not fit into tidy little boxes like dates and quantities etc.

This external corpus also provides rules/guidance that can be used to settle disputes about meaning. A good example is the so-called parol evidence rule[3] which speaks to how disambiguation of meaning can take place. There is also a well developed hierarchy of context information that has been established over the centuries to guide the disambiguation process e.g. the history of how the parties have acted to date (“course of performance”), the history of how they have interacted in the past on other contracts (“course of dealing”), general trade standards/conventions (“trade usage”) etc.

As you can see, there is a vast amount of material and arbitration machinery that sits outside each real world contract but is in effect “pulled in” to each contract by the jurisdiction clause. So much for “self contained” :-)

There is another important sense in which contracts are not self contained and it relates to the component pieces that must exist for a contract to actually exist between two parties in the first place. This is where we will turn to next.

[1] https://en.wikipedia.org/wiki/Smart_contract

[2] https://scepticalprophet.wordpress.com/2012/11/21/spherical-chickens-in-a-vacuum/

[3] https://en.wikipedia.org/wiki/Parol_evidence_rule

The power of combinatorics, in, well, everything

It was late in the morning (around 5:30 a.m.) by the time Master Foo arrived at the training center.

"I am sorry I am late", he said as he sat down. "I had trouble finding Raw Sienna. It was hidden under my meditation box."

The students looked at each other askance from behind the screens of their laptops. "Raw Sienna? What is that and what has that got to do with developing 21st Century Web Applications using mashup technologies?." The students had paid good money to attend this training course and had lugged their laptops up Pentimenti Mountain the night before to be here. Not to mention the fact that they had risen from their freezing tent beds at 5 a.m. to suit Master Foo's schedule.

"Before we begin looking at the details of mashup application development, I would like to draw you a picture", said Master Foo.

From the countless folds in his robes he proceeded to extract a scroll of paper, a small vial of a clear liquid (presumably water), three artist brushes of varying sizes and 6 small tubes of paint.

"It will be a landscape. Please pay close attention to the mixing of colors."

Over the next twenty minutes, Master Foo created a landscape watercolor painting of the view from the top of Pentimenti mountain. It had a brilliant blue sky created with Cerulean Blue[1] for the lighter parts and Ultramarine[2] for the darker parts. Beneath the sky there were many - perhaps dozens of shades of green used for the trees, bushes and grass. As he worked, Master Foo picked up colors one at a time on his brush and mixed them deftly in small plastic containers.

"Master Foo", one of the students asked, "you have used two types of blue and you sourced them directly from individual tubes of paint. Yet, you have used many shades of green but they are all mixed from other colors. Why is that?"

"How many different greens can you count in my picture?", asked Master Foo.

"I cannot count them exactly, there are many."

"How many types of green did you see on your hike up Pentimenti Mountain?"

"I do not know. A countless number I guess."

"Indeed so.", Master Foo replied. "Now tell me, how many types of application do you envisage building on the Web using mashup technologies in your career?"

"A countless number!", blurted one of the students over the top of his iBook.

"Indeed so.", Master Foo replied, grinning as he again turned his attention to his painting.

"Color mixing is a limitless universe of potentiality. Out of these 6 tubes of paint I can make a limitless number of colors given enough time and creativity. By learning how to use each color both on its own, and in combination with the other colors, my color palette is unlimited."

"The true key to expressive power - in any medium including computing - is combinatorics.", he continued. To the relief of the still baffled students, he also switched on his laptop and Ubuntu sprang into life.

"Now tell me," began Master Foo as he logged in, "what is a mashup really? What is its true nature?"

"It is an exercise in combinatorics!", blurted an eager student. "The power of the mashup concept lies in the ability to combine bits of existing website screens into new website screens."

"Yes and no", said Master Foo, grinning again.

"The true nature of a mashup is indeed combinatoric but not at the level of website screens. A mashup that grabs bits of existing website screens and puts them all on the same screen is just a collection of portlets. A mashup is a deeper integration. It involves grabbing data and grabbing functionality from existing websites to create a brand new website whose functionality is more than the visual sum of its component parts."

"If that is so Master Foo", why have you shown us how to paint a watercolor picture?"

"I have done so because it is an excellent illustration of how not to think about mashup Web applications. An anti-pattern by analogy."

"Ah. So you are saying that we should look deeper than the screens. Look at the data and the functionality that needs to be integrated first. Then worry about creating the visuals of a website?"

"Precisely. Unfortunately, very few developers will bother to do that."

The room fell silent.

"What can be done about that sad situation Master Foo?"

"I do not think anything can be done, I'm afraid. After all, it is fun just to paint pictures! That is their great attraction in Web application design and their great limitation. It is important to note that the term 'mashup' is nothing more than a modern twist on the phrase 'application integration' with all that that involves. Until today's army of young web designers realizes that, we will see a lot of pictures being drawn with nothing but thin white paper underneath them."

The room fell silent again.

"Enough of that sadness", said Master Foo, clapping his hands together. "Let us begin our study of the true route to our salvation which will probably be called mashup 2.0 or something similar. REST is a software architectural style for distributed systems[3]..."

[1] http://webexhibits.org/pigments/indiv/overview/ceruleanblue.html

[2] http://webexhibits.org/pigments/indiv/overview/ultramarine.html

[3] http://en.wikipedia.org/wiki/REST

Algorithm - explain thyself!

This is an interesting piece on the opacity of the algorithms that run legal research platforms.

http://www.lawpracticetipsblog.com/2017/08/algorithms-that-run-legal-research.html

Digital machinery - in general - is more opaque than analog machinery. In years gone by, analog equipment could be understood, debugged, tweaked by people not involved in its original construction: mechanics, plumbers, carpenters, musicians etc. As digital tech has advanced, eating into those analog domains,  we appear to loosing some control over the "how" of the things we are building...

The problem, quite ironically, also exists in the world of digital systems. These are regularly redone from scratch when the "how" of the systems is lost, typically when the minds involved in
its original construction - the holders of the "how" - cease to be involved in its maintenance.

With Deep Learning, the "how" gets more opaque still because the engineers creating these systems cannot explain the "how" of the decisions of the resultant system. If you take any particular decision made by such a system and look for a "how" it will be an essentially meaningless, extremely long mathematical equation multiplying and adding up lots of individually meaningless numbers.

In part 15 of the What is Law series I have posited that we will deal with the opacity of deep learning systems by inventing yet more digital systems - also with opaque "hows" - for the purposes of producing classic logic explanations for the operation of other systems:-)

I have also suggested in that piece that we cannot, hand on heart, know if our own brains are not doing the same thing. I.e. working backwards from a decision to a line of reasoning that "explains" the decision.

Yes, I do indeed find it an uncomfortable thought. If deductive logic is a sort of "story" we tell ourselves about our own decision making processes then a lot of wonderful things turn out to be standing on dubious foundations.