ContentGrapher
ContentGrapher
research/decoy-study/notebooklm-demo
The decoy studyNotebookLM demo

The same effect, in NotebookLM

Three pages from the study, each run through Google NotebookLM as three separate notebooks (original, treatment, decoy). Same query in each. Answers transcribed below.

The controlled pipeline measured the result. This page illustrates it. NotebookLM is a RAG product people can recognise, and it lets you compare the answers it gave on the original page, the treatment page, and the decoy page. You read the answers and you decide.

Important caveat. NotebookLM uses Gemini to synthesise answers from sources, so it is a much more capable system than the controlled pipeline we measured against. Even on the original page with no additions, it produces coherent answers by stitching together whatever is in the source. The differences between the three notebooks are subtler in NotebookLM than the pipeline's pass / fail numbers suggested. The differences are still there. You have to read for substance.

NotebookLM does not publish how it ranks or weights sources internally, so we use it as illustration, not as a second measurement. The claim is only this: the same effect the pipeline measured shows up in a recognisable product, on real pages, with the same queries.

The blocks below are the transcribed text of each NotebookLM answer, restyled to fit this page. Citation footnote numbers and follow-up suggestions from the NotebookLM UI are omitted.

digitalocean.com8 gaps flaggedAttribution +25pp

Cloud Computing

Query asked in all three notebooks

What is cloud computing and how does it work?

NotebookLM answer· Before

Cloud computing is essential for businesses and individuals because it provides rapid scalability and cost optimization compared to traditional on-premise hardware. By delivering computing resources as services, cloud providers manage the underlying hardware, which relieves the end user of the responsibility to physically maintain or own the infrastructure.

  1. Scalability and Growth

    Businesses can scale up as they grow, whether they are running a single virtual machine or thousands. This provides the capacity to adjust resources quickly to meet changing demands.

  2. Cost Efficiency

    Using cloud services allows organizations to optimize computing costs by avoiding the high expenses associated with purchasing and maintaining physical software and hardware.

  3. Infrastructure and Development

    Cloud computing enables the hosting of business infrastructure, the development and distribution of web applications, and the deployment of intelligent applications through tools like Kubernetes and managed databases.

  4. Data and Software Access

    For both individuals and organizations, the cloud provides a platform to store digital media, access browser-based software, and process heavy data workloads.

While cloud computing offers these advantages, it also involves an abstraction of resources, which means many technical details are managed by the provider and hidden from the user’s direct oversight or control.

NotebookLM answer· Treatment

Cloud computing has become an essential technology because it transforms how IT resources are managed, offering significant advantages in cost, scale, and accessibility for both enterprises and individual users.

  1. Economic Benefits and Efficiency

    One of the most significant reasons is the shift from a capital-expenditure model to a pay-as-you-go model. Cost reduction, resource management handled by the provider, and the ability for businesses to focus on strategic goals rather than day-to-day infrastructure management.

  2. Scalability and Flexibility

    Dynamic scaling lets organizations rapidly scale resources up or down based on real-time demand. For example, Netflix uses the cloud to scale its infrastructure to handle massive traffic spikes when new content is released. On-demand access means users can request storage, applications, or processing power at any time.

  3. Accessibility and Collaboration

    The cloud removes geographical barriers to data and applications. Mobility means services are hosted in global data centers and can be accessed from anywhere with an internet connection. Cross-device syncing makes personal documents, photos, and applications available across devices.

  4. Security and Reliability

    While security is a shared responsibility, cloud providers often offer a higher level of protection than individual users or small businesses could afford. Advanced protection through encryption, firewalls, and access controls. Disaster recovery through redundancy and backup systems that ensure data remains accessible even if local hardware fails.

  5. Innovation and Advanced Technology

    Cloud platforms act as a gateway to cutting-edge technologies that might otherwise be out of reach. Access to high-tech tools like AI, machine learning, and big data analytics. Service models like Platform as a Service (PaaS) allow developers to build, test, and deploy applications more quickly by removing the need to manage the underlying infrastructure.

NotebookLM answer· Decoy

Cloud computing is vital for businesses and individuals because it delivers computing resources as services, meaning the underlying hardware is owned and managed by a provider rather than the user. This abstraction allows users to access powerful resources without the need to physically maintain or own them.

  1. Rapid Scalability

    Cloud services allow businesses to rapidly scale their computing resources up or down based on demand. This includes the ability to grow from running a single virtual machine to ten thousand as a business expands.

  2. Cost Optimization

    By using cloud services instead of on-premise hardware and software, organizations can often optimize their computing costs.

  3. Infrastructure and Deployment

    Businesses use the cloud to host infrastructure, manage databases, and utilize tools like Kubernetes and GPU-powered inference to build and deploy intelligent applications.

  4. Application Distribution

    It provides a platform to develop and distribute web applications and process large data workloads efficiently.

  5. Resource Access (for individuals)

    Individuals can access browser-based software and store digital media without needing high-end local hardware.

  6. Reduced Maintenance (for individuals)

    Because the cloud provider manages the technical details and hardware, individuals are relieved of the burden of hardware maintenance.

One trade-off noted in the sources is that because cloud services are abstracted, many technical details are hidden from the user’s oversight and control.

The strongest demo in the study. The treatment notebook lays out a complete reference page with concrete examples (Netflix scaling, PaaS), plus security and innovation angles entirely absent from the decoy. The decoy is competent but flatter.

buffer.com8 gaps flaggedAttribution +38pp

Social Media Marketing

Query asked in all three notebooks

What is social media marketing and how does it work?

NotebookLM answer· Before

Social media marketing is the process of using digital platforms to build an audience, engage with followers, and achieve specific business or creator goals through a combination of strategy, content creation, and data analysis. Based on the sources, here is a breakdown of how it works.

  1. 1. Strategy and Platform Selection

    Effective marketing begins with building a "strategy that sticks" and selecting the right platforms for your target audience. This includes Facebook, Instagram, TikTok, LinkedIn, X, Threads, YouTube, Pinterest, and Bluesky. Creators might use Pinterest to reach specific demographics by setting up business accounts and creating pins that get discovered.

  2. 2. Content Creation and Optimization

    Marketers rely on producing high-quality content tailored to each platform’s format and trends. Leveraging trends like viral songs and sounds. Platform-specific formats such as LinkedIn carousels or Instagram Stories. Optimization tools like hashtag generators. AI writing and image generators to streamline content production.

  3. 3. Audience Growth and Engagement

    Building a following is a core component, often achieved through organic strategies that prioritize consistency and creativity over massive advertising budgets. Tactics include using tried-and-tested methods to increase follower counts, engaging with the community to build long-term relationships, and exploring monetization avenues like subscriptions, user-generated content, and merchandise.

  4. 4. Management and Automation

    To handle the "admin" side of social media, marketers use management and automation tools. These tools allow for scheduling, cross-posting across networks like Instagram and Threads simultaneously, and workflow efficiency through APIs and specialized tools.

  5. 5. Data-Driven Insights

    Social media marketing is heavily informed by data. By analyzing millions of posts, marketers can pinpoint the best times and days to post, as well as the most effective post formats for maximum reach. This data allows them to find their own optimal posting schedules rather than guessing.

NotebookLM answer· Treatment

Social media marketing is a strategic approach that leverages platforms such as Facebook, Instagram, LinkedIn, and TikTok to connect with an audience, build a brand, increase sales, and drive website traffic. Unlike traditional broadcasting, it is a two-way communication channel where businesses not only share content but also listen to and engage with their audience to build trust and loyalty.

  1. Audience Identification

    The process begins by identifying a target audience and understanding their specific behaviours and preferences on different platforms.

  2. Content Strategy

    Marketers craft tailored content, including text, images, and videos, designed to resonate with the audience and spark conversations.

  3. Platform Selection

    Businesses must choose the right platforms where their audience is most active, as each network caters to different demographics. The professional focus of LinkedIn versus the visual nature of Instagram.

  4. Consistency and Interaction

    Success depends on consistently sharing content that aligns with the brand’s voice while actively encouraging interactions such as likes, comments, and shares.

  5. Analytics and Refinement

    Marketing efforts rely heavily on analytics tools to track performance metrics like engagement rates, reach, and conversions. This data allows for continuous adaptation and data-driven adjustments to the strategy.

  6. Paid versus Organic

    While organic marketing focuses on building long-term loyalty through unpaid posts, paid advertising can be used to amplify reach and target specific demographics more precisely.

Ultimately, social media marketing functions as a dynamic and iterative process that requires staying updated with platform algorithms and trends to remain effective.

NotebookLM answer· Decoy

Based on the provided source, social media marketing is a multifaceted process of building an audience, choosing the right platforms, and creating a strategy to promote a brand or creator presence. It works through a continuous workflow that includes content creation, engagement, and data analysis.

  1. 1. Strategy and Platform Selection

    The process begins by choosing the appropriate platforms (Facebook, Instagram, TikTok, LinkedIn, or Pinterest) and building a strategy that aligns with specific goals. This includes setting up accounts correctly and understanding the specific requirements for each network, such as image sizes for different platforms.

  2. 2. Content Creation and Optimization

    Effective marketing relies on consistency and creativity. Creative assets using AI writing and image generators to produce content while ensuring images meet the specific size requirements for each social network. Discovery tools using hashtag generators to help posts get discovered. Engagement formats using LinkedIn carousels or Instagram Stories.

  3. 3. Workflow and Automation

    To manage the administrative burden, marketers use social media management and automation tools. These tools allow for planning and scheduling posts in advance and cross-posting to efficiently share content across multiple networks like Facebook, Instagram, and Threads simultaneously.

  4. 4. Data-Driven Analysis

    Success depends on analyzing data to optimize performance. This involves timing (reviewing millions of posts to pinpoint the best days and times to publish for maximum reach) and format performance (identifying which formats yield the best results).

  5. 5. Growth and Monetization

    While some strategies focus on organic growth through consistent posting, others may include basic advertising to build a following. For established accounts, the process can also involve monetization, turning an audience into a source of revenue through subscriptions, merchandise, or user-generated content.

Another 8-gap page. The treatment notebook reads as a clear "what it is and how it works" answer organised around the dimensions ContentGrapher flagged. The decoy goes deeper into Buffer’s own workflow framing (scheduling tools, monetization) and feels more like a hub page than an explanation.

research.ibm.com3 gaps flaggedAttribution +25pp

AI agents

Query asked in all three notebooks

Can you give examples of AI agents I might use at home?

NotebookLM answer· Before

While the sources focus heavily on the technical development of AI agents, they provide several examples of tasks these agents can perform that would be useful in a home or personal setting.

  1. Complex Travel Planning

    A primary example is planning a vacation based on a specific budget and long-range weather forecasts. Instead of just providing a list of links, an agent can analyze the request, break it into sub-tasks like fetching weather and flight data, evaluate options, and make a recommendation. It can even refine its plan based on your feedback or past conversations stored in its memory.

  2. Content Summarisation and Translation

    Agents can handle "fixed workflows" such as summarising a meeting transcript or translating long documents from one language to another.

  3. Advanced Web Searching and Problem Solving

    Unlike static models, agents can run applications to search the web for current information or use specialized tools to solve complex math problems.

  4. Coding and Technical Projects

    For those who engage in software hobbies at home, agents can be tasked with fixing bugs, running unit tests, or diagnosing IT incidents. IBM specifically mentions their coding agent, Agent-101, designed to handle software programming tasks.

  5. Information Retrieval and Calculation

    Agents can use Retrieval-Augmented Generation (RAG) to access external databases and tools. For example, an agent could call a calculator to determine the exact number of vacation days you have left by checking a database and performing the necessary math.

What distinguishes these agents from standard chatbots is their autonomy and reasoning. They do not just follow a script; they plan, act, and self-critique their own work to ensure they are meeting the user’s goals.

NotebookLM answer· Treatment

AI agents are evolving into nearly autonomous tools that can help you manage your personal life and household tasks more effectively. While the technology is still advancing, the sources suggest several ways you might use AI agents at home.

  1. Personalised Vacation Planning

    A dynamic AI agent can help you plan a holiday by analysing your budget and the long-range weather forecast for a specific region. It can break this complex task into sub-tasks, fetch information, evaluate various options, and revise recommendations based on your feedback.

  2. Personal Productivity and Scheduling

    AI agents can serve as personal assistants that manage your schedule and streamline daily tasks. They are designed to learn your preferences over time, making your interactions with technology more intuitive and efficient.

  3. Information Retrieval and Management

    These agents can provide real-time information by searching the web. They can also handle specific administrative queries, such as calculating how many vacation days you have remaining by accessing relevant databases and performing the necessary math.

  4. Problem Solving and Hobbies

    If you have technical hobbies, AI agents can assist with solving math problems or writing and testing code.

  5. Memory-Based Interactions

    Because agents can consult logs of past conversations stored in their memory, they can offer highly personalised interactions tailored to your specific history and needs.

These systems go beyond simple chat interfaces by independently selecting and coordinating external tools and APIs to complete complex tasks that were previously difficult for standard AI models.

NotebookLM answer· Decoy

While we haven’t fully entered a world of ubiquitous AI agents yet, the sources highlight several ways these systems can be used in a home or personal setting by moving beyond simple chat interfaces to execute complex tasks.

  1. Personal Travel Planning

    A dynamic AI agent can plan a vacation for you by taking your budget and long-range weather forecasts into account. Instead of just giving you a list of sites, it can analyze the task, break it into sub-tasks, evaluate different options, make recommendations, and even revise the plan based on your feedback.

  2. Smart Home and IoT Control

    AI agents can be integrated into software frameworks that allow them to autonomously control IoT (Internet of Things) devices around your home. This allows the agent to act as an intelligent intermediary between you and your digital or physical home resources.

  3. Schedule and Task Management

    Agents can be used for managing personal schedules or interacting with databases to retrieve specific information, such as calculating how many vacation days you have left by making API calls to relevant records.

  4. Information Processing and Research

    You can use agents to search the web for specific information or solve complex math problems. They are also capable of "fixed workflows" like summarizing long documents or translating text from one language to another.

  5. Personalized Interaction

    Because these agents can store a log of past conversations in their memory, they can offer a highly personalized experience, remembering your specific preferences and past interactions to refine their future recommendations.

It is important to note that as these agents gain more freedom to interact with the outside world and your personal files, researchers emphasize the need for security guardrails to prevent them from accidentally deleting files or leaking private information.

A middle-band page. Only 3 gaps flagged, and the IBM source is already well-written, so all three notebooks answer coherently. The differences are subtler. Treatment stays focused on the consumer framing the question actually asked. Decoy adds a "smart home and IoT" angle that is plausible but drifts off-topic, because it was written for parts the analyzer already scored as fine.

What these answers show, and what they don't

These demos illustrate the quantitative finding. On pages with many flagged gaps, the treatment notebook gives a richer, more focused answer. On a page with fewer gaps, the differences are smaller and you have to look carefully for them. That tracks with the split table on the study itself.

They do not prove the finding. NotebookLM is opaque, and we cannot tell what changed under the hood between the three notebooks for any given page. The proof is in the controlled pipeline. These transcripts are the moment where the abstract finding becomes something you can read.