The Lifestyle Copilot & Why You Should Be Embedding Intelligence in Your Business Today!

A few weeks ago I had the opportunity to speak at Hampton Roads' premier developer conference, HR DevFest. Thanks to Custom Ink being a great community company, we got to discuss the Rise of AI Copilots using a basic framework to identify what evolutionary stage various AI industries or use cases reside within. The idea shared here is to be intentional as to where we spend your's or your company's time. The left (Applied AI) or the right (Research & Development)?

Obviously if revenue were important, you would leverage applied AI workhorse techniques such as classification, summarization, decision-making, in-context learning, and knowledge retrieval. Today's applied AI has broad internal or external use cases. My favorites are still grounded in HBR's WINS Framework. Everything on the applied side is a great way to add intelligence and value to your organization.

However, sometimes we gotta move from the consumer side of applied AI to a prosumer model where established solutions are scarce. Requiring research and development from both customers and ISVs alike. Fun fact, this is where most of the industry seems to push us 😅. The reasons to operate in this mode are real, there are hard problems we need to solve. Yet it can be risky if you make decisions to operate in this area based solely on demos or a idealized future visions with little to no industry research.

Today's R&D cycles center mainly around Multi AI Agent Systems (MAS), a rebranding of Autonomous Agents from last year. Today we are using tools such as crewAI, AutoGen, and Experts to automate more or build more advanced retrieval systems. However, I think we should step back a bit and examine the value of these systems in the larger context of... The Rise of AI Copilots.

Two Months. Years of Impact!

Before we get too far into our Copilot discussion I'd like to ground us in few months of recent releases from The Big Five + 1 (OpenAI). Starting with the most recent, Apple's WWDC24 event. It is hard to underestimate the cumulative impact of all these announcements.

• June 10th Apple WWDC24 - Siri superpowers with Apple Intelligence. On-device AI using SLMs. Developer integration using App Intents.
• May 21st Microsoft Build - Windows Copilot Runtime, 365, Teams, & Studio. GitHub Copilot Extensions & Workspace. Real Time Intelligence with Fabric. On-device AI using SLMs. Recall.
• May 14th Google I/O - AI Overview in Search with multi-step reasoning.
• May 13th OpenAI Spring Update - Multimodal GPT-4o. Realtime conversations with interruptions. New Mac desktop integration.
• April 30th Amazon - GA of Amazon Q Business & Apps (Preview)
• April 12th Google Cloud Next - Vertex AI's Agent Builder. Gemini for Google Workspace. Gemini Nano in Chrome. Ask Photos.
• April 9th Meta AI Day London - Training 400B Llama 3. Llama 3 w/Facebook, Instagram, & WhatsApp

I have included a few highlights from each event I think are relevant to the conversation. Overall there are two big themes I see unfolding here that everyone should be paying attention to.

1. Edge AI will be deployed at scale to mobile & desktops using SLMs.
2. Integrated Ecosystems adding AI Copilot(s) into their platform(s).

So What Are Copilots?

AI Copilots & the Future of Knowledge Work

How AI copilots are keeping developers longer in a flow state and why AI copilots more broadly could be the start of an industrial revolution for knowledge work.

Andreessen HorowitzKevin Scott, Bob Swan

Let's first take a look at Kevin Scott's article from Sep of last year. It is still very much accurate today. A summary from that post defines a Copilot as:

A Copilot is an AI tool that can assist knowledge workers and developers with their cognitive work and tasks.

Pretty broad description, yet narrow use case. I get the fact Developers have a front row seat to this change we are going through. However, we can do a bit better based on recent events. To help, Kevin made the following points which describe why AI Copilots are important. We can update these a bit along with an example.

1. We are in the "Next Industrial Revolution". Automate!
2. Optimize flow state for Developer Workforce Productivity.
3. More opportunities for Entrepreneurs and SNBs. Ex: Salesforce Survey.
4. Focus on real-world problems. Not the latest hype.

Use Case Copilots

A great example of an AI Copilot for a specific use case is easy. It is one we all use by the same name, GitHub's Copilot. Here are the top 5 ranked by my personal awareness, impact, and usefulness.

1. GitHub's Copilot
2. OpenAI's ChatGPT & Custom GPTs ( Grimoire )
3. Amazon's Q Developer
4. Google's Gemini Code Assist
5. Apple's Xcode Intelligence

Site-Specific Copilots

Another common Copilot can be seen on many of your favorite products or websites. No, I am not talking about the AI ✨ sparkles ( transactional AI integrations ) that you find sprinkled atop existing user interfaces of established products like Canva, Airtable, Notion, or Wix.

What I am talking about are virtual product assistants. A good example of a site-specific Copilot is Shopify's Sidekick.

Shopify Magic and Sidekick: AI for Commerce

Harness the power of AI designed for commerce with Shopify Magic and Sidekick, a growing suite of free AI tools to help you start, run and grow your business.

Shopify

Shopify's Sidekick seamlessly integrates into their seller's platform to facilitate ad-hoc conversational interactions. It answers questions and automates tasks by interfacing with APIs, offering tailored advice and instant support to help manage everything from inventory to sales trends. I think most folks see these as chatbots.

Integrated Ecosystem Copilots

With the recent addition of Apple's Siri using Apple Intelligence and Microsoft's Copilot + PC, we now have direct line of sight into a new breed of Copilots. These Copilots span across Big Tech's integrated ecosystems which includes their edge devices, operating systems, cloud services, and software applications. We can update Kevin Scott's definition too.

An AI Copilot is an agentic system that is omni-present within an integrated ecosystem.

This is our new bar... and better fits what is happening with the industry. The implications are massive! Before we dig into those disruptions, here is a list of the Big Five + 1 (OpenAI) and their nascent Copilots.

• Apple - Siri Drawing on Apple Intelligence
• Microsoft - Copilot - Using Phi-Silica with Microsoft 365 & Studio
• OpenAI - ChatGPT - Using the GPT-4o Model
• Google - Astra - Using Gemini Live with Google Workspace
• Meta - Meta AI - Llama 3 with Facebook, Instagram, & WhatsApp
• Amazon - Q - ??? Model with Q Business & Apps ( Site-Specific )

Integrated ecosystem Copilots will fall under two distinct, yet overlapping categories, Business Copilots and the new emergent Lifestyle Copilot. Together each will drastically change how we strategically integrate AI in our products.

The Battle for Context

I used to think larger and larger context window sizes were going to settle who had the better Large Language Model (LLM) and hence value. Instead I should have been thinking what frontier model builders have the most context, the most data (business or lifestyle) to ground AI's responses. More importantly, the capabilities to deliver AI inference and integrations to business & consumers alike. Yet customer chatbots like Amazon's Q still dominate in our industry. Which leads me to wonder ...

Who wants to jump around on the internet talking to dozens of virtual assistants? The future should be personalized AI Agents that act on our behalf.

This is not a new take. Yann LeCun, Head of Meta’s AI Division, explained in his interview on Lex Fridman's podcast about the idea of AI assistants not just retrieving information but also acting as intermediaries, filtering and possibly taking actions – once we master real-world hierarchical planning.

If you buy into this idea, as I do, our next question should be what Integrated Ecosystem AI Copilots (virtual assistants) have the most context and capabilities? Obviously Google, Meta, AWS, & Microsoft have our data, yet I think two of the Big Five now stand out among the crowd, Microsoft and Apple.

Introducing Copilot+ PCs - The Official Microsoft Blog

An on-demand recording of our May 20 event is available. Today, at a special event on our new Microsoft campus, we introduced the world to a new category of Windows PCs designed for AI, Copilot+ PCs. Copilot+ PCs are the fastest, most intelligent Windows PCs ever built. With powerful new silicon capable of an incredible…

The Official Microsoft BlogYusuf Mehdi

Introducing Apple Intelligence for iPhone, iPad, and Mac

Apple today introduced Apple Intelligence, the personal intelligence system for iPhone, iPad, and Mac.

Apple Newsroom

For later comparison, i'll focus on a few shared criteria when looking into each of these ecosystems.

• The primary role of the AI Copilot.
• Where is AI inference happening.
• The modalities of the on-device inference.
• Personalization capabilities & knowledge grounding.
• Ability to act autonomously on a users behalf.

Google Misses on Copilots?

Before we dig into Microsoft & Apple, let's take a closer look into the recent Google Cloud Next '24 event in April.

This is Thomas Kurian introducing Google's new Vertex AI Agent Builder. Right before he teed up this introduction of Mercedes Benz CEO, Ola Källenius.

"Mercedes Benz is working with us on customer agents to help people in their amazing cars."

From there Ola, goes on about customer service agents on the website (yawn) all during a backdrop of in-vehicle motion graphics suggesting an AI Agent in their vehicle could do all sorts of things like turning off the AC, rolling the windows down, and turning the music volume up. A "series of actions" presumable in one step with AI? He closed with Google's partnership as key for their next level navigation and automated driving. Matthew Berman was quick to call out

"This is the biggest missed opportunity I have ever seen. Why isn't there an agent built into the infotainment system in the Mercedes? That seems like the most obvious use case..."

Of course he is right. Though I do think there is a good reason for the missed use case. Mercedes, like any other car maker, loves to own the console. Yet Apple has never needed to build cars to be good at CarPlay thanks to their platform leverage. Which allows Apple to capitalize on its existing software ecosystem, hardware integration capabilities, and massive user base. They will consistently do this.

Google is certainly winning in other areas. Gemini for Google Workspace is going to be a massive workforce multiplier. In the lifestyle category, AI additions such as Ask Photos will provide meaningful value.

Microsoft (Business+Lifestyle)

So let's dig into our first personalizable Copilot in an integrated ecosystem, Microsoft's Copilot+ PCs along with all the other relevant product announcements from Microsoft Build 2024.

Copilot with GPT-4o will be able to leverage real time voice conversations using live screen sharing. The demo showcased a user in Minecraft with Copilot helping them work through game mechanics & objectives. In real time! Huge Lifestyle potential here if Microsoft can run a world knowledge model locally. More so if the novelty of these AI demos can transition to real-world interactions.

On the Business side, we have Microsoft Copilot Studio, an AI agent builder that can personalize Copilot on your PC or external customers. Copilot Studio provides familiar tools such as instructions, triggers, knowledge, actions, and interactive HTML elements with their Adaptive Cards framework. It’s built for enterprise productivity. Like in a good way. It integrates seamlessly with Microsoft 365 & Teams and connects to data sources like Salesforce or Jira. You can also deploy your Copilots to numerous channels, including Slack. But the biggest news is in their orchestration toolchain, to quote Satya Nadella.

... Copilot will work in the background... you're going to have Copilots plus Agents with this async behavior. You can delegate authority to Copilots to automate long running business processes ...

Clearly that is going to be a huge win for office productivity. Assuming we think humans are doing the work? 🤔

The real news from Build is not how well they can integrate with OpenAI or Azure, but how well they can run AI inference locally. To do that Microsoft unveiled their new Phi Silica Small Language Model (SLM) running on their new NPU silicon. Using Windows Copilot Runtime, developers can build applications leveraging Phi Silica or other ONNX models to read or summarize text, chat, perform tasks for math and code, or even generate images. Locally!

However, I dug into the Microsoft local AI frameworks. They feel disorganized and overlapping. I found no semantic database SDKs or unified Windows system integration. Microsoft seems to have focused on an outside-in development method. So much so, Recall was implemented using screenshots. I'm seeing very inside-out ecosystem integration. They may be relying on business developers to do the heavy lifting. 🤷

Apple (Lifestyle+Business)

Our second integrated ecosystem Copilot has a name, Siri. Now embedded with AI super powers thanks to Apple Intelligence announced a WWDC24 a few weeks ago.

At the heart of this ecosystem is a novel architecture. Apple has developed a new frontier Small Language Model (SLM) shared through a fine-tuned adapter architecture. This enables on-device intelligence to create images and provide system-wide writing tools for highly personalized use cases, such as photo search, smart email notifications, writing tone adjustments, and context-aware actions across devices.

Thanks to the App Intents framework, developers can tap into the same semantic index and orchestration that powers Siri, making their app’s content and actions discoverable with System Experiences, Shortcuts, and Intents. In typical Apple fashion, solid Human Interface Guidelines support this across the entire integrated ecosystem, including macOS, iOS, and watchOS.

Apple's developer documentation is very well organized. I was able to find SDKs for on-device Vision and semantic embeddings. At a glance I can see that security is implemented from the bottom up. Only Siri or the user can orchestrate across applications. My only question is would companies like Slack have incentives to move their semantic indexes to the client to leverage App Intents and empower Siri with more context or will they keep everything server side? Either way, Apple seems to be the clear Lifestyle Copilot winner here.

Just like CarPlay, Apple will demonstrate that you do not need to own enterprise IMAP or SMTP servers to win at the INBOX.

Counterpoints, I've never felt that Apple's Messages for Business really took off. Despite their partner list, I can not recall one time where I interacted with the product. What I can say is that any company that nails the human-in-the-loop AI-powered virtual assistant... just may just end up being the best decision platform for AI-enabled business.

The Future is (SLM) Slim

In only a few short months we went from companies like VMWare & Broadcom pushing CPU-based inference using quantized models for local, server, and edge compute to direct line of sight into the success of SLMs with powerful models such as the Phi family.

We are going to explore SLMs more in a future article. Please do signup up for my newsletter for updates. Only a few emails each month.

The Bottom Lines

First think Siri will be the Lifestyle Copilot winner. Last to market wins! Sure, execution eats strategy, but do not mistake a lack of visible progress that Apple does not execute well on strategic vision. They do. You just missed it. We should Let Tim Cook! Why? The numbers are clear on this one, iOS will soon take over Windows in the operating system category. Combined with macOS, Apple's Siri operating system market share might be unstoppable.

Source: StatCounter Global Stats - OS Market Share

Remember, at lot of industry analysts got it wrong during the WWDC24 event. OpenAI is not as integrated as folks think. Apple is using its own models. Eventually OpenAI will be ROKR'ed out of the platform.

Copilot Criteria Rubric

Here is a simple rubric on the high level criteria we were tracking for the context battle of the Copilots. I encourage you to dig into the frameworks and learn more. There could be a lot of opportunity here for a few killer apps that fill in the gaps early for users.

AI Text & Image Commoditization

On-device SLMs powering both text and images should be a warning sign for companies like Notion, Canva, and Grammarly that have AI price and packaging strategies. Why pay these companies for + subscriptions when every Apple device includes powerful, built-in writing tools and smart text processing capabilities? The AI Sparkle™ ✨ pattern of adding value may wane soon.

It was easy to miss this point during the WWDC24 event. Every, every, every text input will be have AI-enabled writing tools. This quote comes directly from the documentation.

If you’re using any of the standard UI frameworks to render text fields, your app will automatically get the ability to use Writing Tools.

Build Intelligence Now!

My prediction is that we have 6 months tops from the release of these platforms before Copilots really start looking outward to integrate with the internet on your behalf. Remember the Yann LeCun points from above, AI assistants will not just retrieve information but also acting as intermediaries on our behalf. Once that starts to happen, site-specific chatbots are going to look be less useful.

Just as Apple built their Intelligence System, your company should too. Here is their same diagram with my recommendations to focus on.

Consumers are going to demand semantic integration with your business or product. Do you have the right Infrastructure and APIs to meet them where they (or their virtual assistants) are?