Virtual Assistants vs Virtual Advocates

Friendly RobotExisting Virtual Assistants include Siri, Google Now, and Cortana (others?). But the idea has been in around in computing and science fiction since forever (ie: the 50s).

In my opinion, the current commercial vision of the Virtual Assistant is deeply flawed. This is a consequence of the current consumer computing environment, which in recent years has become incredibly centralised to a few powerful companies, and very closed, dominated by walled gardens that don’t allow programmatic access to the important parts of their platforms.

Here’s a cool vision of Virtual Assistants from Apple in the 80s

Below, I talk about these flaws as I see them, and about what an alternative type of agent, a “Virtual Advocate”, would do instead.

Briefly, a Virtual Advocate will be like a Virtual Assistant in many ways. You talk to it in natural language, it responds in natural language. You ask it high level things: “Arrange a meeting with George next Wednesday”, “Are there any changes to the wikipedia page on Ferrets since yesterday?”, “Cut off my subscription to Fnergling Monthly”. But it is based on open standards, talks to an ecosystem of cloud based Apps built on open protocols, there are potentially many alternative implementations of Advocates available across platforms that talk to the same ecosystem, and the user is in control. 

This is preliminary thinking, designed to get a conversation going.

Not Search, but the Browser

This is a deep mistake being made in the first generation of assistants.

The companies making these assistants seem to see them as analogous to Search (aka Google). They are thinking of them as the place people come to ask questions. They reason that if they can provide the answer right there and then, the user wont leave the company’s sphere of influence, perhaps landing in a competitor’s space instead. The oligopolists look at Google’s domination of search, and their ensuing ability to own advertising, and think hey, we could wrest away / continue to own that space in the next generation.

But people don’t go to Google to find things out / do things as a first point of contact. They go to their phone or their browser. And what can they reach from this point? Other people, things made by other people, the entire virtual world. Which includes Google et al, but isn’t Google; Search is still just another app.

A Virtual Advocate that people actually want to use will be an entry point to an open universe of content and functionality made for and by everyone. It will talk to the next generational equivalent of a web page (something like a cognitive app), which will be built on open standards; most importantly on an open protocol.

Note protocol, not platform. A platform is something a closed commercial entity builds to control and contain a space. A protocol describes how a set of disparate entities can cooperate in building an open space. Facebook is platform, the web is protocol.

The web exploded, in the face of stiff commercial opposition, because it was based around an open protocol. Anyone could build a browser by implementing a published protocol, and, more importantly, anyone could build a website by implementing a published standard (or using a webserver based on one). No one had to ask anyone’s permission. If you wanted to build a site, and someone else wanted to visit it, no one had to run it by Tim Berners Lee to see if that’d be ok.

Virtual Advocates will be the same. They will talk to Apps on the behalf of users; these apps will be defined by an open behavioural specification. Sign up to the app in a standard way, pass in a query in a standard way, get a response in standard way, notify the user of new points of interest in a standard way.

Part of this standard could be the semantic web, but that’s not enough. There needs to be a whole protocol for programmatically talking to clever Apps.

And there can be many, many takes on an Advocate. An advocate is anything that can mediate between users on the one hand, and apps on the other. It could be a fairly stupid thing that doesn’t do much, or it could actively protect the user from the worst excesses of the apps it wrangles, keeping a large no-mans-land between the apps and the user. It could be a natural language interface combined with a bit of visual webbiness. Or it could be the next generation terminal / command line, a smart shell.

Advocates could be cloud based/backed but don’t necessarily need to be. Need to store data / state? There’s an app for that. They may need a url. They will likely be platform agnostic.

Smart, but not allowed to solve real problems

The virtual assistant can notice meetings in your email. It can tell you things about your schedule. It can prompt you about your favourite TV show. 

Big deal.

To my mind, the biggest problem in personal computing right now (from the point of view of actual users) is overbearing, mind numbing complexity. There are so many services, apps, sites, platforms jostling and fighting to get their slice of our attention, and they don’t play nice. 

So what do we spend our days doing? Hunting through all these things for what we need. What’s that notification about, did I already see that somewhere else? What’s the website for my train, and is it warning me of interruptions in service? Where’s that document I made last week; what tool did I even use to make it? What’s Gerry’s email address, or wait, do I talk to him through whatsapp? Why am I looking at my phone… I’m sure I meant to do something, but now I’m in my facebook stream. If I install this app, will it do something evil? Should I put my email address into this website?

Meanwhile, AI is on the way. The next wave of computing is to Take X and add AI. Watson APIs promise all kinds of high level AI services; as do others (eg: google’s prediction API). But there’s no way for a user to expand a virtual assistant’s capabilities with specific services built on top of these new APIs (or indeed on machine learning techniques, etc).

Currently we’re on a path toward apps that use these techniques, and big data, to manipulate us in more eyewateringly powerful ways than ever before, while we, like babes in the woods, have no defence. Unaltered humans cannot successfully use these technologies and also not be predated on by them, we require serious algorithmic backup to help us out.

An Advocate should be able to use the tools we already use, on our behalf. Just reading webpages for us would be a boon: “Are there any disruptions to my train listed on the rail company’s website?”, “How many people does wikipedia say live in Iceland”, etc. Plus more active uses: “Sign me up to a trial account for that new social network”, “Get me off that mailing list”.

To achieve these things, the advocate should use composable modules, the “Apps” I referred to above. These apps would be the equivalent of the modern website, but be for Advocates to use rather than for people to use directly. As they are for Advocates, they need to be machine accessible, not human accessible. So, apps can be less like silos (as modern websites are) and more like small, composable tools, like the linux shell uses. Higher level apps can use lower level apps. Also think dependency injection.

These apps, small composable tools, would advertise what they do, and how to ask them to do it. “I can parse english into a parse tree”, “I can take a parse tree and annotate it with alternate vocabulary”, “I can figure out roughly what an annotated parse tree is about”, “I can take an annotated parse tree asking about the content of a website, look up the website, and answer your question about it”. 

As AI services and techniques continue to improve, the apps available to advocates can become more and more interesting. An open, composable architecture for fullfilling user requests using clever combinations of apps will make the best use of these emerging technologies, and harness them in our interests, rather than letting them run wild against our interests, by strictly mediating our interactions.

Not made for us

The assistants are being built with the oligopolistic tech companies’ interests in mind rather than those of their users. The usual design approach is employed in which just enough value is delivered to the user to keep them engaged (superfical things about flight times, when movies are on, travel time to somewhere), and then the bulk of the work is in “scaling” – ie: gathering as many of us in as possible and keeping us coming back over and over.

The services are closed (the user can’t expand them and can barely customise them), they deal in trivialities and shopping, and they are structured so that great corpuses of data are gathered about individuals that the individual herself is entirely walled off from. That’s an architecture we take entirely for granted now, apparently barely worthy of note, but I find it important.

Increasingly, regular users understand the Faustian bargain of the computing devices and environment we are being offered. We understand that these services are manipulative and serve their corporate owners, and we hope we can eke out more value than we lose, in combination with feeling that we must use these services because of the negative side of network effects; ie: everyone else is using them, so if we don’t, we are on the outside of important networks.

The devices we own and use now are a combination of skinner box and shopping mall. No longer are they about productivity enhancement and personal empowerment; they are about superstimulus and manipulation. Devices have appstores, apps have in-app payments. Apps scream for our attention, and feed us small, frequent, random positive rewards for our attention, so we can’t look away. The manipulation professions (marketing, advertising, …?) have flooded online and reach their fingers out toward us through our devices.

Advocates will be the new browsers / phone appstore + launcher. But unlike browsers, they will mediate strongly between us and the frenemy apps, helping us get use from apps without being violated. They might carefully limit the data being sent to abusive apps, warn us of potential abuses, notice problematic notification behaviour, decide to prioritise competing information from competing apps in ways the user wants rather than ways a platform owner or app provider wants.

Advocates should be able to vary wildly in behaviour, as long as they ultimately talk to the apps the user chooses to subscribe to. This should encourage experimentation and innovation, and as a side effect penalise closed implementations (why go there if there are open, inspectable, trustable alternatives?).


The modern personal computing environment is more powerful than it’s ever been, but also a bit dystopic and depressing. The current phase, the mobile+cloud phase begun by the iPhone, that killed the optimism of Web 2.0 and replaced it with the shiny shopping malls of Steve Job’s vision, needs to end. AI is coming, and if we don’t correct course, it’s going to be used for manipulation and control and be a tool of disenfranchisement and disillusionment that’ll make commercial television look like flower power.

I believe one part of a swing back could be to promote the open, powerful computing environment of Virtual Advocates + Apps described above, with this smart, truly user centred software component assisting people with the ballooning complexity of the sociotechnological environment and protecting people from the depredations of the other players (particularly commercial ones).

We’re very early on in the Agents phase, probably too early. But it’s the right time for us (common or garden rebel coders) to begin talking about this, maybe even to begin trying to bang out some horrible prototypes. Even really stupid Advocates that you have to say painfully specific things to could be useful if a few good apps pop up (eg: stuff that reads webpages for you, stuff that can handle signups, logins, unsubs for websites/services). Or if not useful, then interesting. What more could you ask?

Great thanks to John Hardy and Jodie O’Regan, who’ve listened to my ranting and helped with ideas.

Virtual Assistants vs Virtual Advocates

Free online books on singing for primary students

My darling wife has released two books she’s written on singing for primary students; a voice workbook and a theory book. They’re available under Creative Commons licenses on her studio blog.

Free online ebooks on Singing, Sight Singing and Music Theory

If you find yourself teaching singing or theory in a classroom or choral situation, and need some framework, these are for you.

Free online books on singing for primary students

Infinite Bookshelf


I’m looking at a bookshelf and wondering why it exists.

We love to collect books. They’re a symbol of knowledge and learning. Reading is fun, and to look at a bookshelf is to remind yourself of that feeling of being absorbed by a tome, lovely. We have a sensory relationship with books, their smell, their texture. That kind of thing is an emotional connection to a fondly remembered past shared by many readers, an olfactory mnemonic, functioning to remind us not so much of specific readings but of the positive emotions associated with them.

Loaded as they are with meaning and triggers and symbols, they also function as powerful units of consumption. We like to own them. Really they’re excellent in terms of consumer commodity; we feel that owning them will make us happier and or more satisfied in some way (smarter! more knowledgable! more interesting to others!), we feel like their acquisition advances us in some material way toward some perfect ideal of the intellectual life, and we can never have enough of them, because they are barely fungible. It is impractical to own a copy of every book, so there is always something more we haven’t got, something more to lust after and think that if we only had that one, maybe we’d be happy.

These books are the material object, the embodiment. These are the things we can hold in our hand, put on a shelf, throw at someone.These are the things we cache, that we hoard.

There is another book, very different from the book described above. That book is the content, the information separate from the embodiment. It is the ideal platonic book. It is the class of things of which the embodied book is an instance.

When we desire knowledge, it’s an inherently disembodied thing. We want to get it into our minds, and the way it gets there is more or less immaterial*. Also, we want to talk about it with others, and again its form is mostly not interesting.

More or less generally, it is the case that a mind with more knowledge is more desirable than one with less. The infinite regress of this says that one with everything is best of all. So there’s been a dream probably forever, of accessing all human knowledge, and the extension of that via the golden rule, to the concept of universal access to all human knowledge. Everyone should be able to know anything they desire to know.

Wonderful recent inventions to this end have been digital information encoding, and the internet. Cory Doctorow says “The Internet is the greatest copying machine ever invented, and it’s never going to get any harder to copy stuff.”

But let’s get back to the realm of books. Should we wish to, we now have the ability to trivially provide access to all the books ever written and ever to be written to anyone who can get online, and indeed a lot of them already are available in this way.

(Of course many of the books are not online, and wont be for some time. Unfortunately, we have a system in place which controls what platonic information can be available via the great copying machine, and it jealously guards many of the books. It’s about making money, and it is entirely invested in the vision of the book as embodied instance, not as platonic class. They are not selling the platonic book; they are selling wodges of paper with whatever they can decorate it with to make you want to buy it. I don’t really want to go on about that here, except to declare that if we have to take sides, I’m on the side of universal access to all human knowledge. Given the great copying machine, there is an inherent selfishness in any act of restricting access (which includes requiring payment and certainly includes requiring it to be printed on dead trees). For this act to be moral, the benefit gained must outweigh the penalty paid by the entire mass of humanity due to lack of access. A personal gain would have to be huge indeed in this instance. Or, is it just that the owners believe their works to be of such marginal value that excluding the great mass of humanity is barely a cost at all? But I digress.)

Let’s just imagine that everything worth considering is available electronically in some form, for free. To a large extent in fact this is true now; I find I can get by for the most part with freely available information. I feel certain this is a continuing trend, and that as time progresses it will become more and more practical to live this way.

What does this mean for my bookshelf?

It depends a lot how you feel about books. Are books their platonic class, or their embodied form?

If they are the embodied form, then this free availability of (zero cost digital copies of) the platonic form is beside the point. How can I experience the book properly without the paper, the feel of the cover, the smell of the print? How can I make my notes, put in a bookmark, turn over a corner? What about the little rip on page 83 that reminds me of the day all those years ago when my little brother took the book without asking? What about the experience of a library full of row upon row of books? What about a comfortable study, a warm fire, an overstuffed chair, and a shelf of good books?

These are undeniably things of beauty, but it is largely nostalgia. It relates to past culture, and experiences of early life. Subsequent generations have their own culture and nostalgia, and these things you value might mean no more to them than reminiscences of the ice man on horse and cart, or the sound of enslaved africans singing in the cotton fields.

I remember in the mid 90s, arguing with a friend about the merits of adding simulated analogue hiss to a digital recording. His position was that it makes the sound more natural; I just thought it was a cultural artifact, learned by listeners and by sound engineers in their formative years due to the analogue equipment of the era. I predicted future audiophiles, not yet born perhaps, would crave that “ringing” sound of lower bitrate mp3 encodings. Now, more than 10 years later, I read about a survey that “shows increasing preference for MP3 by youngsters.”

So it will be with books. We pine after physical books, but our kids and their kids, well, who knows? Perhaps an aging early generation Kindle, plastic yellowing, reading surface scuffed, will ignite in them the same sensations of intellectual warmth? Or will their eyes tear up ever so slightly at the sight of the splash screen for Adobe Acrobat Reader?

If on the other hand the platonic form is for you the book, then the future is bright. Google’s unassuming search page becomes the door to the thieves’ treasure, and Aladin need only click “search”. Suddenly, the need to cache is gone! If you can believe that the internet will continue, that the mass of information available will only grow, then there is no need to keep a copy, even a digital one. If you need it, you get it, read it, and if you had to download a copy you can just delete it afterwards, because if you need it again in the future, you’ll just go get it again. I find this to be enormously liberating. A cache, a collection, a hoard, is an anchor. You have to devote time and resources and mental energy to its creation, maintenance, management, protection, and it is always imperfect. To suddenly be freed from the need is something difficult to describe; it’s like the feeling of getting rid of all your possessions and just driving. It is weightlessness.

There is something important in the embodied form point of view, though. Whatever the main screen of Google might be, it is both more and less than a bookshelf. That search screen ignites in us the promise of infinite unexperienced treasures, but it says nothing of those treasures we have already experienced, of what has informed us and made us who we are. It is not personal.

This also applies to you if the platonic form is your (perfect eternal) cup of tea. You might be able to access all the books via the net, but can it tell you what you have already read? What about the books you’ve studied, vs those you’ve skimmed, vs those that have been recommended to you but you’ve never gotten around to looking up?

I think this an important function of a bookshelf; it is a history. Any bookshelf is the representation of the thought process of the person who put the books there. This might be the librarian’s structured set of books in various categories, or it might be the haphazard personal collection of somewhat random books collected over decades.

For the digital realm, I’ve been trying to replace this with a fairly minimal effort, my reading list. It’s only a short list so far, but it’ll grow, and eventually become unwieldy. I guess it can go through the natural stages of organisation, from list to hierarchy to search enabled mountain.

But is this the right way to go? I feel as though something more is required. How do we keep a usable history of our access of information, which integrates well with normal use of the web, which scales? What secondary effects could be realised through collecting and organising this information well?

This idea of intellectual history is important I think. It is important because we forget ourselves. We forget what we have read, or we remember it only partially, or incorrectly, or both of these things. We find that we hold opinions which were arrived at by chains of reasoning, but no trace of the chain remains. So over time, without our history, what we considered a mighty intellectual fortress becomes a curiously ornate shell, elaborate, but thin walled and empty.

In that light, the bookcase looks less impressive to me. It was never really fit for that purpose.

* apologies

Infinite Bookshelf

Nice fonts

We largely use Open Office at home (one machine still has an old copy of MS Office, but I’m trying to convince the little guy who owns it to try OO.o). Recently Jodie was looking for more fonts (for her windows machine) and we ended up here:

1001 Free Fonts

It looks like a bit of a scam site, but it’s actually very cool, and the fonts are lovely. Well worth a visit. It has good installation instructions too, iirc.

(btw to get these to show up in Open Office, you need to close open office, and all related apps, and reopen it. If you don’t get the splash screen with the progress bar when you open it, then something was still running, try again. If in doubt, log out and back in, or just reboot.)

Nice fonts