The Coming Age of the Context Engine

People talk a lot about information overload, but I think the worst problem we have in information management today is memory overload -- the inability of the human brain to retain all the important information we run into in our careers.  There's more stuff we need to remember than you can possibly hold in your head.  The more successful you are, the more information you need to remember -- and the worse the problem becomes.

I think what we need is a context engine, an app that helps you recall the context around any bit of information in your life.  Unlike a search engine, a context engine indexes just the information in your life, and supplements your own memory.  "How do I know this person?"  "What's the agenda for my next meeting?"  "Who sent me that article last year, and where the heck is the article?"  A context engine will help you answer these questions quickly, anytime and anywhere you need the information. 

The product that I'm working on, Zekira, is a first generation context engine.  In this post I'll discuss the need for a context engine, how it would work, and our status with Zekira.  I'll also give some information on how you can help, if you're interested.


The trouble with information overload

Information overload is a hot topic with a long history.  The term was coined in the 1960s, and popularized in 1970 by Alvin Toffler's book Future Shock, according to an excellent article in Wikipedia (link).  But the idea goes back further.  Xerox implies that it invented information sharing through the development of the photocopier in the mid-1900s (link), And there have apparently been complaints about too much information for as long as we've had writing.  The Bible complains about the proliferation of books, the Romans worried about it, and so did the ancient Chinese.  Once Gutenberg got going with movable type, the complaints increased (link).

Information overload is a popular subject online.  The Wall Street Journal said Google returned 2.92 million hits for it in 2009; the same search today returns 3.76 million, an increase of about 770 references per day.

Prominent authorities opinig on the hazards of information overload include the New York Times (link), Wired (link), and none other than the big consultancy McKinsey, which says it is "killing productivity...and making us unhappy" (link).

The critics of information overload complain that it bombards people with so much data that they are stunned into stupidity.  They become low-grade data zombies, incapable of making intelligent decisions.

The answer, we're told, is to take in less information.  The experts tell us to delete e-mails and limit our exposure to information online so that we can reserve time for thinking deep thoughts and forming long-term memories.

Okay.  It makes sense that we should set aside time to think.  But I believe distraction isn't a function of how much information you bring in, it's a function of how much self-discipline you lack.  There's always something you can distract yourself with; if it's not e-mail it'll be Angry Birds.  You could have the same problem if you had five e-mails a day or five hundred.

I think blaming "information overload" for the problem of distracted people is like blaming "water overload" for the problem of drowning.  The fact is, modern society runs on the flow of information.  The more information you can handle, the more productive you'll be, and the further you'll go in your career.  Given the way the economy works, telling people to limit their information flow is a little like telling them to make themselves stupid.  Instead, I think, we should be increasing our ability to manage that information, so we can be more productive.


The real problem is memory overload

Once you step back from demonizing information itself, it's easier to identify the problems that we have in dealing with so much information.  I think the biggest problem in information management today is the limitation of human memory. 

Think about how you remember things.  It's usually through associations -- I saw it in the newspaper when I was at that cafe, I read it in that article on The Register while I was riding the bus, etc.  When people have more information to remember than their brains can hold onto, those chains of association start to break down.  You remember the fact that you once knew something, but can't recall the information itself.

As I talk with busy knowledge workers -- the type of people who manage the most information -- I hear stories about half-remembered information all the time.  You'll see a person and know that you've met them, but can't recall the details about how you know them or what you discussed with them.  Or a topic will come up and you'll remember that you read something important about it, but you won't recall where you saw it or how you could find that information again. 

Often you know the information is stored somewhere on a computer or smartphone or website, but you have no way to look for it in the moment you need it.  Even if you remember to look up the information later, it's usually extremely hard to find, and you can't take the time to do it. 

The more successful you become in your career, the more information you have, and the more overloaded your memory gets.  Of course it eventually overflows.  The problem is so ubiquitous that most of us don't even think of it as a problem; it's just a feature of life.  We shrug it off as a "senior moment" and uneasily move on.

But it has nothing to do with age; it's a function of experience.  Take all the information held by a mid-career professional and stuff it into a 20-year-old's head and he or she will have the same problems. 

When you add up all those "senior moments" across all the people they happen to, they constitute a huge loss in productivity among the busiest and most pivotal people in the economy.  The only reason we tolerate this situation is because we assume there's nothing we can do about it. 

But I think we can.  The combination of mobile technology, low-cost computer storage, and web services makes it possible to build what I call a context engine -- an app designed specifically to help you recall the information in your life, and all the context around it.

You'll use a context engine to quickly recall:
    -All the details of your relationship with someone -- how you met them, messages and documents you've exchanged, and meetings you've been in together.
    -The backstory to a meeting, including the messages that led up to it, attendees, notes and pictures you took during the meeting, and followup messages afterward.
    -A tweet or Facebook post or e-mail you saw months ago mentioning a great new restaurant that you want to try.
    -That report sent to you five years ago by some guy you met at a half-remembered conference in Boston.


How the context engine will work

A context engine needs to do three things with your information:  Collect, connect, and communicate.

1. Collect.  To build a map of all your information, the context engine needs to gather it from all the places where your information is stored.  That means, first, scanning the hard drives and other storage devices connected to your personal computer.  E-mails, contacts, and meeting records all need to be extracted from whatever messaging and calendar system you use.  For most mid-career professionals, that means digging into old Microsoft Outlook archives, called PST files.  Other documents -- especially presentations and word processing files -- need to be sucked in as well, along with the most ubiquitous file format in business and academics, the PDF.

But you can't stop with the PC.  The context engine needs to reach out to your web apps, to extract things like gMail messages, tweets, and Facebook posts and contacts.  And the information on your smartphone needs to be included, everything from contacts to text messages to pictures.

This process should be automatic and comprehensive.  Everything should be indexed.  You shouldn't be asked to choose which files you want to remember, because you can't know in advance what you'll need.

2.  Connect.  Once all that information has been collected, it must be organized.  That means indexing it not just by keywords, the way we would for a traditional web search, but by all of its attributes, including date, time, location, type of content, and so on.

This is a key difference between a web search engine and a context engine.  In web searches, we look almost exclusively for keywords, and we use the wisdom of crowds to determine which matches are most important.  That works great for searches of publicly-available content, but it breaks down when searching your personal archive.  You may not remember the right keyword for a document or message, and the wisdom of crowds is much less useful for ranking results, because everyone's context is unique.  Instead, a context engine needs to offer many search paths through the archive, so people can search using whatever bits of information they do remember about a topic.

The context engine should also present information to you in a way that lets you jump between bits of related data.  Say you're looking for the record of a lunch meeting.  You might be looking for it because you want to find the name of the person you met with, or some messages you exchanged with that person.  Or maybe you just want the name of the restaurant so you can eat there again.  All of that information needs to be cued up so you can jump to it easily.  Again, the goal is to help you re-create those half-remembered chains of association. 

Many of the products that in the past have tried to organize personal information (such as Google Desktop) have mimicked the keyword-centric searching we do on the web.  Web search is so ubiquitous that we're all a bit like the man with the proverbial hammer -- every problem looks like a nail.  But I think personal context requires a radically different structure to the database and UI.  It's not about searching for things, it's about navigating through your context.

3.  Communicate.  You don't know when you'll need to remember something, so the context engine needs to be available on your mobile devices.  In particular, I think a context engine is a killer app for tablets in business.  Imagine always having your entire information history at your fingertips in every meeting and every conversation.  How much more productive could you be if you had a perfect memory all day long?

I can't tell you how many people in Silicon Valley have told me sheepishly that they don't know what to do with their iPads at work.  They generally love them at home, where they access entertainment and informational content.  But at the office, particularly in meetings, they tend to turn into tools for covertly checking messages and browsing when the meeting gets slow.  Please don't misunderstand, I know there are many things you can do with an iPad.  But I'm reporting what I hear from a lot of iPad users.

Far be it from me to judge the way others fill their time, but I think the context engine would give you a good business reason to carry your tablet all day.

That means the database needs to be hosted in the cloud, which creates all sorts of important security challenges.  Having your extended memory hacked is utterly unacceptable.


Building the context engine

As you know if you've been following my earlier updates, the startup that I'm working on, Zekira, is building a context engine.  The company consists of four engineers plus myself, and we've been working on it for more than a year.  Zekira is the fulfillment of a dream for us.  One of us, Rudi Diezmann, has been working on personal search products since the 1980s.  Others of us first thought about this problem when we were working at Palm almost ten years ago.  We were looking at user problems a PDA or smartphone could solve, beyond helping you manage your calendar and contacts.  There was a group of customers who responded very strongly to any product that could help them recall information and the context around it.

But only recently have mobile computers and wireless networks become powerful enough to let you build a full-function context engine. 

The first version of Zekira is in early beta.  It runs on Macs and PCs, and right now it indexes information found on your computer and any storage attached to it.  Our goal is to take Zekira mobile, and to add web data sources, as soon as possible.  But we did the first version on personal computers so we could get started testing the database and search capability.  Besides, there are a lot of people with old Outlook and Office archives who would be happy to turn a context engine loose on them.

Zekira gives you a little search window that you can leave up on the screen, or minimize: 

After you do a search, your results appear in this window:

 The four stacks in the center show you all the items that matched your current search.  In this case, we're seeing things related to Tom Shannon, including documents that he wrote or that mention his name, messages you've exchanged with him, and his contact record.  Click on any of those items and you'll see information related to them.

The tabs on the left are filters that let you narrow the search.  Currently they let you search by time/date (the filter shown), name, word, document type, and folder: 

 You can combine multiple filters to do complex layered searches.  The filters are extensible, and we plan to add additional search tools in the future.

We're doing a crowdfunding campaign for Zekira on the funding website Indiegogo.  If you don't know how crowdfunding works, people can make small financial contributions to a project and receive benefits in return, such as a discounted copy of the program when it's finished.  Supporters of Zekira can also get access to the beta version of the program, and listing as a sponsor in the about box of the finished app. 

Corporate sponsors of Zekira can get advertising here on Mobile Opportunity, a unique offer since I don't generally accept ads (except for one tiny Google ad that gets me access to Google's excellent traffic monitoring tools).  The advertising sponsorship offer is a great way for a company that has a little bit of advertising budget left at the end of the quarter to help itself, and also help support a great product.  The ad offer is limited to three companies, and is first-come, first served.

If you'd like to learn more about Zekira, you can visit our crowdfunding site here, and our website here.  And here's a video of Zekira in action:



If you have feedback and suggestions for Zekira, I'd welcome your comments.  And if you like the idea, please help spread the word about our crowdfunding campaign.  The more support we get, the faster we can move on the project.

No matter what you think of Zekira, I hope you'll agree that the time is right for a context engine.  With that and an info pad, I'd be one very happy camper.