In this final installment of how to build an NLA (a domain-specific or DS-NLA) — the how informing the who question (that we started with back in mid-December 2023) — I focus on what may be the most overlooked set of questions about how a system should be built — not just for serving users, but supporting it’s operators. To repeat with my closing of the previous installment: it’s not a “one-and-done” process to build an NLA; the builders need to become operators; those who will, henceforth, refine and extend the DS-NLA over time.

Remember the “Pit Crew?”

Let’s return to the earlier Formula One (F1) metaphor (from midway through Part 2) where each NLA is being custom built, including, but not even remotely limited to, a few select off-the-shelf parts. We must recognize that not even Max Verstappen could finish a race without the Pit Crew.

Just as you wouldn’t expect Max to drive a real race without a Pit Crew, you wouldn’t want to turn the public loose on a domain specific NLA — especially for a topic as sensitive  as elections — with nobody “minding the store” so-to-speak.

In the case of a domain specific NLA (DS-NLA), there could be several different kinds of people in the this type of “Pit Crew.” Perhaps the most familiar would be IT staff who set-up and monitor the virtual servers that run the NLA software itself; such virtual servers managing tasks such as:

• Ensuring uninterrupted content served to users’ web browser or App; 

• Making service calls to separate computer servers that run the actual SLM (Small Language Model);  and

• Providing the equivalent of an “in-flight-recorder” of how the NLA is working.

It’s not a terribly useful tool if nobody is monitoring the system to notice and respond when it becomes bogged-down due to high user demand, or network latency to the SLM, or other factors.

But speaking of the separate servers that field service requests of a base model, who are the people running those? 🤔 In the typical case of a NLA built on today’s typical “GIGA-LLM,” that would be the staff of Tech-Titans generating enormous revenues from running these behemoths. However, for more domain specific NLAs using other base models, it’s not that clear.

Then there is likely a need for humans to handle on-going assessments of how the DS-NLA is or isn’t working well; for example:

• Occurrences of out-scope responses;

• Growing rates of non-responses;

• Insufficient or deficient citations; and even

• Dreaded hallucinations. 😳

Whatever the safety measures are that an NLA might have, they’re surely never going to be perfect, and will need improvement or “adjusting” over time. That’s why the “in-flight-recorder” function is necessary if not imperative to:

• Provide information to human reviewers who can flag potential problems; and also very likely to

• Provide data for analytics tools that human reviewers use to manage the scaling of operation.

Changes Within the Domain

One type of iteration on the NLA-building process is incremental improvement in the system, independent of the domain-specific information. However, that authoritative information base can also change over time.

As a result, another requirement is for facilities to refresh the corpus of authoritative information that the NLA is drawing from in generating responses. While election law was once seen to be a stable body of authority with only periodical changes, today’s hyper-partisan polarized environment is creating considerable instability in election law, regulations, policies, protocols, and resulting precedures. Accordingly, the only constant is change (in election laws at state levels). Thus, in the domain-specific world of elections information, “refactoring” becomes a real probability. Therefore, at some point, an elections DS-NLA is going to need to be refreshed with an updated knowledge base and better filters. And here’s the real catch: who is going to monitor to collect the input for such a refresh, and who actually performs the refresh itself? That’s the DS-NLA “Pit Crew.”

And there are likely other Pit Crew roles, as well as other questions about just who fills these roles. I argue that the above examples illustrate some serious uncharted territory. And “time’s a-wastin’” to get started exploring it… 🤓

So, Who Best to Make and Crew a DS-NLA?

That finally brings us all the way back to the Who question…

I hope by now it should be evident that making an election domain-specific NLA (which is a universe apart from “Chatbots”), like that we’ve envisioned in this article series here, is going to require an interdisciplinary team of people with domain expertise, design expertise, typical technical expertise, and some technical expertise that’s not well understood today.

There is important work to do in at least six areas:

• Base models and safety;

• Methods for building the (data and information) corpus;

• Functionality for trustworthiness;

• Usability issues and UX design;

• Technical design for deployment (including scale);

• Infrastructure for operators; and

• Those things you’re going to remind me I’ve overlooked here. 🤔

To be clear, creating such an agent is simply is not credibly going to be from one or two people with a “kewl” idea and an OpenAI subscription (or worse, someone or two with a mendacious idea to create the most chaos-causing friendly-looking but malicious elections NLA ever encountered).

I’m not gonna lie, it’s a somewhat daunting prospect as my colleague explained at a gathering of philanthropists last October (although it is increasingly tractable with solid architectural considerations we’re now considering). Yet, like many such innovation challenges, the only way to fully understand what’s needed is … to just get started.

There are many building blocks and lessons-learned ready to be used. The basic requirements are clear. And, at least insofar as election administration and democratic electoral processes, I think that the need for such a system is clear.

Elections’ time and tides wait for no one, so now that we’ve taken a look at the how and the who of building an Elections NLA, we’re charting a course and likely to soon set sail. My parting shot on that: All aboard! 😎

Sincerely, I hope this triggers some conversation (comments are open below; DMs are open; eMail is open). We’re going to have a whole bunch more to say about this in the coming weeks and months. We’d surely appreciate your input.