In partnership with
MediaMorph Edition 106 - by Mark Riley
Bot a lot - publishers need a door policy
Was this newsletter forwarded to you? Sign up here
The written-by-a-human bit
Nightclub bouncers have been doing it for years. They stand behind a velvet rope and only wave through the punters who look sober enough, rich enough or good-looking enough - or who have membership and are on a list. Waving everyone through is not a good look, nor is a very long line around the block.
Savvy publishers now need to work out and enforce their own door policy as bot traffic overtakes human traffic on the web. According to Cloudflare Radar data from late April 2026, bots now account for 57% of all HTTP requests on the open web. Humans make up the remaining 43%, as reported by Media Copilot.
AI agents are driving the shift. Before the generative AI era, bot traffic accounted for around 20% of web activity, with Google's crawler the largest single source. Now, agents performing tasks on behalf of users generate requests at scale. Cloudflare’s CEO, Matthew Prince, put it plainly at SXSW: a person shopping for a digital camera might visit five websites, while an agent doing the same task might hit 5,000.
The president and CEO of the News/Media Alliance, Danielle Coffey, has raised a red flag: "By illegally scraping our content, repackaging it, and giving it to consumers without adequately directing them to our sites, AI companies are using our own content to undermine our businesses."
According to the alliance and Tollbit, it reads your work, summarises it for a user, and sends almost nobody back. Cloudflare's own analysis says publishers now face roughly ten times the difficulty winning the same traffic volume from Google's AI Overviews as from its classic search results. From OpenAI, the multiplier is 750 times. From Anthropic, 30,000.
Theo Bamber, the newish president of the News Media Association in the UK, makes a similar point in The Times:
“Worse, their latest play is to bury the visibility of links to trusted, verified news sources underneath AI summaries, choking the traffic that keeps newsrooms alive. And what is Westminster doing about it? Dithering.”
To be fair to Westminster, the Competition and Markets Authority has since weighed in, asking Google to allow publishers to opt out of AI Overviews altogether. This is a well-intentioned intervention to try to level the playing field, but full-on boycotts rarely work.
The dilemma for publishers is figuring out which bots to let in (search crawlers, discovery bots, shopping bots, research bots) and on what terms. Only then can they follow the data and assess a workable door policy.
Playing nicely and politely asking the bad bots not to come in does not work either. Robots.txt is a polite request. Named bots from reputable companies may honour it. Spoofed user agents and bad actors ignore it entirely.
For years, the web ran on a simple exchange: crawlers copy your content for search, and search sends you traffic. That bargain made sense when Google was the main bot at the door. It makes less sense when thousands of agents arrive, extract value and refer almost no one back.
Publishers who treat all automated traffic alike, either waving it all through or blocking the lot, will leave money or discovery on the table. The better move is a door policy: identify the bots, separate discovery from training from abuse, set terms for each, and enforce them technically (using AI, of course).
And on what terms? Dmitry Shishkin has published a thought-provoking piece on LinkedIn about agent readiness and how to meet agent needs, as well as human needs and wants.
“AI agents are (or will be soon) acting on behalf of users - retrieving, summarising, comparing and delivering information before a person ever reaches a publisher's article. The unit of value is shifting from the document to the verifiable fact. How journalism is discovered, produced and distributed — all of it must change.”
Multiple toll booth startups, as well as Cloudflare, are trying to crack the problem, but most take their own percentages in the process.
B2C is not over yet, but there is much at stake. Media execs need to be asking the questions - what is our bot door policy? How do we enforce it? How do we keep the ugly drunks out?
Mark Riley, CEO Mathison AI
AI and Journalism
This week’s best articles, as chosen by HANA and our editors
Cloudflare CEO: Bots have overtaken human traffic online The Media Copilot - June 5, 2026 In a historic shift, bots now account for 57% of web traffic as of April 2026, surpassing human users, while representing an alarming 94% of all login attempts and highlighting the escalating sophistication of cyber threats. Organizations must rapidly enhance their security measures to combat this surge in automated attacks and protect sensitive information. |
Meet the journalists training the AI models that might replace them Reuters Institute for the Study of Journalism - June 5, 2026 As AI technology reshapes the journalism landscape, many journalists are transitioning into roles that involve training and assessing AI systems, finding both new opportunities and ethical dilemmas in the process. While some view AI as a threat to traditional media jobs, others see it as a collaborative tool that can enhance productivity and allow reporters to focus on more impactful storytelling. |
Wayne State students explore AI’s impact on journalism, public relations, and media careers Today@Wayne - June 8, 2026 Wayne State University's hybrid course "Artificial Intelligence and Communication Work," taught by Dr. Rahul Mitra, delves into the transformative impact of AI on communication professions while fostering critical thinking about ethics and authenticity. The course features industry guest speakers and a service-learning project with the Detroit Rescue Mission Ministries, providing students with practical insights into navigating the complexities of AI in communication. |
New York Legislature Passes Landmark Bill to Disclose AI-Generated News to the Public Nysenate - Senator Patricia Fahy and Assemblymember Nily Rozic have joined forces with the NY FAIR NEWS Act coalition to unveil new developments in legislation aimed at enhancing media transparency and combating misinformation. The NY FAIR NEWS Act seeks to promote equitable representation and protect the public's right to accurate information, although specific provisions were not detailed in the announcement. |
SECOND COURSE ADDED: AI Without the Guesswork: Smarter Journalism Without Prompting and Hallucinations Gijn - Our curated courses are fee-based offerings that help sustain our team and trainers, ensuring we can continue to provide free, high-quality resources to our community. By balancing accessibility with financial sustainability, we enhance the quality of our educational content while compensating our dedicated trainers. |
These 16 new journalism jobs could help publishers “future-proof” their newsrooms The media landscape is evolving with exciting job opportunities across renowned organizations like The Economist, The New York Times, and Bloomberg, as they seek innovative leaders in AI engineering, product design, and audience engagement to leverage artificial intelligence in journalism. Key roles include senior positions focused on enhancing newsroom technologies, developing strategic off-platform engagement, and integrating AI solutions to transform how audiences interact with news. |
Leading Canadian news organizations join SPUR’s global coalition to shape the future of AI and journalism Editor and Publisher - June 4, 2026 Major Canadian media organizations, including CBC/Radio-Canada and The Globe and Mail, have joined the SPUR Coalition, an international alliance focused on establishing standards for intellectual property rights in relation to generative AI. This initiative aims to ensure fair compensation for quality journalism while reinforcing the importance of trusted news in combating disinformation and supporting democracy. |
AI, journalism and the uncertain future of the public square The Seattle Times - June 4, 2026 The rapid rise of AI, exemplified by ChatGPT, raises urgent concerns about its impact on journalism and democracy, as major tech companies increasingly exploit news content without compensation, threatening the sustainability of original reporting. As leaders from over 60 countries convene to address these challenges, the need for responsible AI use that supports rather than undermines creative industries is more crucial than ever. |
Talking Local Journalism, AI and the Power of Community with Algenon Cash — DavidsonLocal.com DavidsonLocal.com - June 3, 2026 Davidson Local, a trusted source for hyperlocal news founded by Antionette Kerr, emphasizes the irreplaceable value of human connection in journalism amid the rise of artificial intelligence. As they navigate the evolving landscape, their commitment to community-centered storytelling and authentic engagement remains essential for fostering informed local dialogue. |
A New Startup Lets Rich People Use AI to Target Journalists Novara Media - June 2, 2026 Objection.ai, a controversial AI startup backed by billionaires Peter Thiel and Balaji Srinivasan, allows wealthy individuals to challenge journalists over unfavorable stories for a fee, raising concerns about creating a "parallel justice system" that undermines journalistic integrity. Critics warn that the platform, which employs LLMs to evaluate complaints, could stifle whistleblowing and public interest journalism, serving as a "high-tech protection racket" for the powerful. |
UK Regulator Staunches Google’s AI Content Grab Tech Policy Press - June 5, 2026 The UK's Competition and Markets Authority (CMA) has mandated Google to give publishers control over the use of their content in AI-generated search summaries, ensuring clear attribution and a No Retaliation rule that protects search rankings. This landmark decision highlights the need for regulatory measures to address the challenges posed by AI technologies on journalistic content, as publishers face immediate losses before compliance takes effect in December 2026. |
Where to Invest $100,000 Right Now, According to Experts
Investors face a dilemma. When the S&P 500 finished its worst quarter since 2022 last month, diversifiers like bonds and bitcoin fell too.
Even with the turnaround in mid-April, analysts at Goldman Sachs and Vanguard have projected low-single-digit annualized returns from 2024-2034.
Bloomberg asked where experts would personally invest $100,000 for their March monthly edition.
One answer that surfaced for a second time? Art.
It's what billionaires like Bezos and the Rockefellers have privately used to diversify for decades.
Why?
Appreciation. The ArtPrice100 Index outpaced the S&P 500 overall from 2000 to 2025
Low-correlation. The postwar contemporary segment has moved independently of traditional investments like stocks since ‘95.*
Resilience. A scarce, physical, and global asset class with decades of demonstrated demand.
Thanks to the world's premier art investing platform, now anyone can invest in works featuring legends like Banksy, Basquiat, and Picasso, without needing millions.
Shares in new offerings can sell quickly but...
*According to Masterworks data. Investing involves risk. Past performance is not indicative of future returns. See important Reg A disclosures at masterworks.com/cd.
AI and Academic Publishing
This week’s best articles, as chosen by HANA and our editors
Wiley Acquires Emerald, Expanding Research Scale and Deepening Proprietary Content Across the AI-Driven Knowledge Economy Wiley has expanded its journal portfolio to around 2,500 titles, reinforcing its leadership in key academic areas like economics, business, and finance. This strategic growth enhances access to high-quality research, positioning Wiley as a dominant force in the academic publishing landscape. |
University of Phoenix researchers publish study examining doctoral students' attitudes toward AI chatbots and ChatGPT use in higher education Prnewswire - June 6, 2026 A recent study from the University of Phoenix, published in the International Journal of AI in Pedagogy, Innovation, and Learning Futures, explores graduate students' attitudes toward AI chatbots like ChatGPT, revealing that favorable views correlate with higher usage rates and highlight the need for clear ethical guidelines in academia. The research underscores the importance of adapting policies to maintain academic integrity across disciplines as AI continues to reshape educational practices. |
University World News cited in thousands of publications University World News has emerged as a vital source in global higher education, with over 5,700 citations across academic publications and policy documents since its launch in 2007. Recognized for its open-access model and diverse commentary from scholars worldwide, it plays a crucial role in shaping research and discussions on pressing issues like governance, digital transformation, and sustainability in the sector. |
Oxford Law Pro wins Gold for two SSP EPIC Awards Oup - Oxford Law Pro has won Gold in two categories at the 2026 EPIC Awards, highlighting its commitment to excellence and innovation in legal resources. This recognition cements its status as a leader in enhancing legal research and practice, showcasing its dedication to user satisfaction and continuous improvement. |
Technological Forecasting and Social Change Paper Development Workshop Successfully Held at Xi’an Jiaotong Xi'an Jiaotong-Liverpool University - June 5, 2026 The recent Paper Development Workshop for “Technological Forecasting and Social Change” at Xi’an Jiaotong-Liverpool University gathered over 50 researchers to enhance their academic writing and navigate the publishing process, featuring insights from Elsevier's Publishing Director on innovative strategies in scholarly publishing. Participants engaged in meaningful discussions on artificial intelligence, digital innovation, and more, with follow-up support promised for manuscript revisions. |
This newsletter was partly curated and summarised by AI agents, who can make mistakes. Check all important information. For any issues or inaccuracies, please notify us here
View our AI Ethics Policy
