e277bdbb1e fix(web-extension): read the playing video on YouTube Shorts (#3463)
## Summary

When the user opens YouTube Shorts after a normal video in the same tab,
YouTube moves the previous video to a paused mini player. The extension
then described the mini player video, not the Short.

- The content script found the video with
`document.querySelector('video')`. That is the first video of the page,
and on Shorts it is the mini player.
- The title and channel came from the watch page heading. That heading
also belongs to the mini player video.
- Only the URL was the URL of the Short.

So the video context had the title, channel, subtitles, progress, and
paused state of the mini player video. The character said that the video
was paused and talked about the wrong video.

This PR adds `pickVideo` in `src/content/pick-video.ts`. A playing video
wins. Among the rest, the largest visible video wins. A video without
size is hidden and never wins. On Shorts pages, the content script does
not read the watch page heading or channel. The title then comes from
the page title of the Short.

The video elements on a Shorts page after a normal video, measured in
the browser console:

| Element | Size | State |
| --- | --- | --- |
| Mini player with the previous video | 400×225 | paused |
| The Short that plays | 315×560 | playing |
| Prepared player for the next Short | 0×0 | paused |

## Change map

| Module | Before | After | Description |
| --- | --- | --- | --- |
| `src/content/pick-video.ts` | None | `pickVideo` | Picks the video
that the user watches |
| `src/content/index.ts` video lookup |
`document.querySelector('video')` |
`pickVideo(document.querySelectorAll('video'))` | Used for the video
context and the vision frame |
| `src/content/index.ts` `findVideoTitle`, `findChannelName` | Read the
watch page heading on every YouTube page | Return nothing on Shorts
pages | The title falls back to the page title of the Short |

## Boundaries and risks

| Invariant or boundary | Failure mode | Protection | Evidence or gap |
| --- | --- | --- | --- |
| The context describes the video that the user watches | The first
video is a paused mini player | Playing video first, then the largest
visible video | Unit tests. Manual test on YouTube Shorts |
| Hidden players do not win | A hidden player is first in the DOM |
Videos without size are skipped | Unit test |
| A normal watch page keeps working | The only video is not picked | One
visible video is always picked | Unit test. Manual test on a watch page
|
| Cost of the lookup | `attach` runs on every DOM change and now reads
the size of each video | A page has few videos, three in the test above
| Not measured |
| The user plays the mini player on a Shorts page | The mini player
video wins, but the title is the title of the Short | None | Not
handled. Rare case |
| Channel on Shorts | The channel is unknown | Unknown is better than
the channel of the mini player video | I did not find a stable selector
for the Shorts channel |

## Verification

```text
pnpm -F @proj-airi/airi-plugin-web-extension test:run   # 2 files, 7 tests passed
pnpm exec eslint plugins/airi-plugin-web-extension      # passed, no output
pnpm --filter @proj-airi/airi-plugin-web-extension build:firefox   # passed
```

Manual test (Zen Browser, temporary add-on from `.output/firefox-mv2`,
AIRI desktop v0.15.1): open a normal video, then open Shorts in the same
tab.

| Check | Before | After |
| --- | --- | --- |
| Live Preview title | Title of the mini player video | Title of the
Short |
| Live Preview channel | Channel of the mini player video | Unknown |
| Scroll to the next Short | Title does not change | Title changes |
| Ask the character about the video | "It is paused", about the mini
player video | The Short that plays |

Not verified:

- Bilibili and other sites. They use the same `pickVideo`. A page with
one visible video behaves as before.
- Chrome build.

This PR does not depend on #3461. They change different files.

I found this problem while testing #3461. I used an AI assistant to
trace the cause in the code and to write the changes. I reviewed the
changes and tested them myself.

---------

Co-authored-by: leafyy <wuqizq@outlook.com>
Co-authored-by: Neko <neko@ayaka.moe>
2026-11-20 10:41:40 +08:00
2026-05-16 17:27:26 +08:00
2026-11-13 03:53:10 +07:00
2026-08-26 20:13:10 +08:00
2026-08-26 20:13:10 +08:00
2025-11-30 20:27:08 +08:00
2024-12-03 16:25:03 +08:00

Project AIRI

Re-creating Neuro-sama, a soul container of AI waifu / virtual characters to bring them into our world.

[Join Discord Server] [Try it] [简体中文] [日本語] [Русский] [Tiếng Việt] [Français] [한국어]

AIRI - A container of cyber living souls, re-creation of Neuro-sama | Product Hunt moeru-ai%2Fairi | Trendshift AIRI - #1 repository of the week across all languages | Trendshift

Heavily inspired by Neuro-sama

Tip

On Windows, you can also install AIRI with winget:

winget install MoeruAI.AIRI

Or install AIRI with Scoop:

scoop bucket add airi https://github.com/moeru-ai/airi
scoop install airi/airi

On macOS, you can install AIRI with Homebrew Cask without adding a custom tap:

brew install --cask airi

Warning

Attention: We do not have any officially minted cryptocurrency or token associated with this project. Please check the information and proceed with caution.

Note

We've got a whole dedicated organization @proj-airi for all the sub-projects born from Project AIRI. Check it out!

RAG, memory system, embedded database, icons, Live2D utilities, and more!

Tip

You can translate AIRI on Crowdin, or open a pull request that changes packages/i18n/src/locales/. Both ways update the same translations.

Do not submit a translation that is the same as the English text. Review rejects it. A string without a translation shows the English text.

Crowdin | Agile localization for tech companies

Have you dreamed about having a cyber living being (cyber waifu, digital pet) or digital companion that could play with and talk to you?

With the power of modern large language models like ChatGPT and famous Claude, asking a virtual being to roleplay and chat with us is already easy enough for everyone. Platforms like Character.ai (a.k.a. c.ai) and JanitorAI as well as local playgrounds like SillyTavern are already good-enough solutions for a chat based or visual adventure game like experience.

But, what about the abilities to play games? And see what you are coding at? Chatting while playing games, watching videos, and is capable of doing many other things.

Perhaps you know Neuro-sama already. She is currently the best virtual streamer capable of playing games, chatting, and interacting with you and the participants. Some also call this kind of being "digital human." Sadly, as it's not open sourced, you cannot interact with her after her live streams go offline.

Therefore, this project, AIRI, offers another possibility here: let you own your digital life, cyber living, easily, anywhere, anytime.

Recent DevLogs & Stories

What's So Special About This Project?

Unlike the other AI driven VTuber open source projects, アイリ was built with support of many Web technologies such as WebGPU, WebAudio, Web Workers, WebAssembly, WebSocket, etc. from the first day.

Tip

Worrying about the performance drop since we are using Web related technologies?

Don't worry, while the Web browser version is meant to give an insight into how much we can push and do inside browsers and webviews, we will never fully rely on this. The desktop version of AIRI is capable of using native NVIDIA CUDA and Apple Metal by default (thanks to HuggingFace & the beloved candle project), without any complex dependency management. Considering the tradeoff, it is partially powered by Web technologies for graphics, layouts, animations, and the WIP plugin system for everyone to integrate things.

This means that アイリ is capable of running on modern browsers and devices and even on mobile devices (already done with PWA support). This brings a lot of possibilities for us (the developers) to build and extend the power of アイリ VTuber to the next level, while still leaving the flexibility for users to enable features that require TCP connections or other non-Web technologies such as connecting to a Discord voice channel or playing Minecraft and Factorio with friends.

Note

We are still in the early stage of development where we are seeking out talented developers to join us and help us to make アイリ a reality.

It's ok if you are not familiar with Vue.js, TypeScript, and devtools required for this project, you can join us as an artist, designer, or even help us to launch our first live stream.

Even if you are a big fan of React, Svelte or even Solid, we welcome you. You can open a sub-directory to add features that you want to see in アイリ, or would like to experiment with.

Fields (and related projects) that we are looking for:

  • Live2D modeller
  • VRM modeller
  • VRChat avatar designer
  • Computer Vision
  • Reinforcement Learning
  • Speech Recognition
  • Speech Synthesis
  • ONNX Runtime
  • Transformers.js
  • vLLM
  • WebGPU
  • Three.js
  • WebXR (checkout the another project we have under the @moeru-ai organization)

If you are interested, why not introduce yourself here? Would like to join part of us to build AIRI?

Current Progress & Roadmap

Capable of

  • Brain
  • Ears
    • Audio input from browser
    • Audio input from Discord
    • Client side speech recognition
    • Client side talking detection
  • Mouth
    • Multi-provider voice synthesis, including ElevenLabs, Microsoft/Azure Speech, OpenAI-compatible TTS, Alibaba Cloud Model Studio, and local Kokoro TTS
  • Body
    • VRM support
      • Control VRM model
    • VRM model animations
      • Auto blink
      • Auto look at
      • Idle eye movement
    • Live2D support
      • Control Live2D model
    • Live2D model animations
      • Auto blink
      • Auto look at
      • Idle eye movement

Development

For detailed instructions to develop this project, follow CONTRIBUTING.md

Note

By default, pnpm dev will start the development server for the Stage Web (browser version). If you would like to try developing the desktop version, please make sure you read CONTRIBUTING.md to setup the environment correctly.

pnpm i
pnpm dev

Stage Web (Browser Version at airi.moeru.ai)

pnpm dev

Stage Tamagotchi (Desktop Version)

pnpm dev:tamagotchi

A Nix package for Tamagotchi is included. To run airi with Nix, first make sure to enable flakes, then run:

nix run github:moeru-ai/airi

NixOS

Electron requires shared libraries that aren't in standard paths on NixOS. Use the FHS shell defined in flake.nix:

nix develop .#fhs
pnpm dev:tamagotchi

Stage Pocket (Mobile Version)

Start the development server for the capacitor:

pnpm dev:pocket:ios --target "<DEVICE_ID_OR_SIMULATOR_NAME>"
# Or
CAPACITOR_DEVICE_ID_IOS="<DEVICE_ID_OR_SIMULATOR_NAME>" pnpm dev:pocket:ios

Quote the target: simulator names such as iPhone 16 Pro contain spaces.

You can see the list of available devices and simulators by running pnpm -F @proj-airi/stage-pocket exec cap run ios --list. @capacitor/cli is declared in apps/stage-pocket, so cap does not resolve from the repository root.

If you need to connect server channel on pocket in wireless mode, you need to start tamagotchi as root:

sudo pnpm dev:tamagotchi

Then enable secure websocket in tamagotchi settings/connections.

Documentation Site

The documentation site lives in moeru-ai/airi-docs.

Publish

Run bumpp to update the monorepo version:

npx bumpp --no-commit --no-tag

Support of LLM API Providers (powered by xsai)

Sub-projects Born from This Project

flowchart LR
  subgraph Apps[Stage applications]
    Web[stage-web]
    Desktop[stage-tamagotchi]
    Pocket[stage-pocket\nexperimental]
  end

  subgraph Shared[Shared product packages]
    StageUI[stage-ui]
    Domain[core-agent\ncore-character]
    Audio[pipelines-audio]
    Renderers[stage-ui-live2d\nstage-ui-three and renderers]
    SDK[server-sdk\nserver-shared]
  end

  subgraph Channel[Desktop server channel]
    Runtime[server-runtime]
  end

  subgraph Integrations[Source-configured integrations]
    Discord[discord-bot]
    Minecraft[minecraft-bot]
    Other[Telegram, Factorio, MCP, and others]
  end

  subgraph Hosted[Hosted backend]
    Edge[Caddy]
    API[api-server]
    Auth[auth-server]
    Database[(PostgreSQL)]
    Cache[(Redis)]
  end

  Web --> StageUI
  Desktop --> StageUI
  Pocket --> StageUI
  StageUI --> Domain
  StageUI --> Audio
  StageUI --> Renderers
  StageUI --> SDK
  Desktop --> Runtime
  SDK <-->|server channel| Runtime
  Discord --> SDK
  Minecraft --> SDK
  Other -. optional .-> SDK
  Edge --> API
  Edge --> Auth
  API --> Database
  API --> Cache
  Auth --> Database
  Auth --> Cache

Similar Projects

Open sourced ones

Non-open-sourced ones

Project Status

Repobeats analytics image

Acknowledgements

Supporters

Thank you for supporting Project AIRI through OpenCollective, Patreon, and Ko-fi.

Project AIRI supporters

Special Thanks

Special thanks to all contributors for their contributions to Project AIRI ❤️

Star History

Star History Chart
S
Description
💖🧸 这是一款自托管游戏,你可以拥有“Grok Companion”这个平台,里面包含了各种老婆角色以及网络生活元素,它们被引入到我们的世界中,旨在达到“Neuro-sama”的高度。该游戏支持实时语音聊天、Minecraft 游戏以及 Factorio 的玩法。同时,该游戏支持网页版、MacOS 版和 Windows 版。💖 🧸Self hosted, you owned Grok Companion, a container of souls of waifu, cyber livings to bring them into our worlds, wishing to achieve Neuro-sama's altitude. Capable of realtime voice chat, Minecraft, Factorio playing. Web / macOS / Windows supported.
Readme MIT
1.1 GiB
0 Stars 1 Watchers 0 Forks
Languages
TypeScript 75.6%
Vue 19.6%
C# 2%
GDScript 1.2%
JavaScript 0.7%
Other 0.7%