11-19 更新 实时
首页/ 技术栈/ 正文

Social Department 构建文化感知视频发现

技术栈 来源:AWS Machine Learning 2026-11-19 03:27 0 阅读 机器翻译
Social Department 构建文化感知视频发现

娱乐营销团队要在制片厂、电视网和流媒体应用之间管理数万条视频素材。大多数营销活动都始于同一个视频发现难题:在素材中找到那一个能引起观众共鸣的瞬间。当需求简报下达,倒计时就开始了,而他们需要的东西,几乎从来不是数据库能够描述出来的。

语义搜索本身存在天花板:它只能回答你能够清楚表述的问题。让它找一个拿着咖啡杯的角色,它会在上千集里找出每一个咖啡杯。而真正驱动娱乐营销活动的那些查询,完全不是这个样子。它们是关乎判断力的问题:一部剧里的哪个瞬间会传播开来,哪句台词值得引用,哪一帧画面能当表情包。

这些问题没法输入搜索框。要回答它们,需要搜索索引不具备的两样东西:一是针对特定内容库,对“值得引用”或“适合玩梗”意味着什么作出编辑层面的定义;二是对内容库之外什么正在流行保持持续刷新的认知。在过去,回答这些问题意味着要让一个对剧集了如指掌的人花上一周时间看素材,而这样的人无法覆盖整个内容库。

在本文中,我们介绍娱乐营销 AI 操作系统 Social Department 如何着手把这些编辑判断类问题编码成可执行的检索流水线。成果就是 QuickSearch——一组单次操作即可完成的查询,无需营销人员描述需求,就能返回他们要找的片段:Shareable Scenes、Quotable Quotes、Memeable Moments 和 Trend Align。

该解决方案中的模型运行在 Amazon Bedrock 上,因此客户数据始终留在 AWS 数据边界之内。数据不会与模型提供商共享,也不会被用于训练底层模型。

Social Department 对负责任 AI 的做法是:AI 提出建议,由人来决定。在生产环境中,代码会将生成的输出与源素材进行核对(例如,如果某句引述不在该集节目的文字记录中,就会被丢弃)。Amazon Bedrock Guardrails 帮助筛查客户搜索文本,以防范提示词攻击,而客户团队中的人员会在每一条帖子发布前进行审批。

Social Department 掌握流水线架构、检索系统以及每项动作背后的编辑标准。AWS Premier Tier Services Partner(AWS 顶级服务合作伙伴)TrackIt 按照这一设计完成实施与交付。Amazon Bedrock 上 TwelveLabs 的 Marengo 3.0 和 Pegasus 1.2 模型为该流水线提供了所依托的场景级视频推理能力。AWS 提供基础设施、模型访问(包括 Anthropic 的 Claude 系列模型),以及用于检索的 Amazon OpenSearch Service(采用其无服务器选项 Amazon OpenSearch Serverless)。

关于范围的说明:本文介绍的是该系统中涉及 AWS 的部分:摄取与索引流水线、检索路径,以及运行这些环节的服务。Social Department 自有的组件,包括其媒体与数据主干、内容分析与发现引擎、编辑标准以及评估计划,仅在理解该架构所需的程度上加以说明。下文提到的具体服务正是本文所讨论的对象,而非 Social Department 整个系统的清单。

Social Department 是面向娱乐营销的 AI 操作系统,服务于各大制片厂、电视网、流媒体平台和 FAST 应用。它能在客户的全部内容库上自动完成营销活动策略、素材创作、发布和效果优化。这些功能大多首先依赖于在素材中定位到正确的时刻,而语义片段检索从一开始就是该产品的核心。

娱乐营销受制于两个约束条件,通用的搜索层无法满足。

营销人员可以描述一个物体、一个地点或一句台词,却无法描述什么内容会传播开来。某个片段能打动观众、另一个不能——这种判断才是这份工作真正的技能所在。它从来都无法自动化,因为它取决于了解观众的行为方式,而不是视频里有什么内容。只理解内容的搜索层需要由人来提供意图,而要想把意图提供好,就得先看过这部剧。

再强的视频理解能力也无法告诉你,观众今天早上在谈论什么。这类信息产生于内容库之外,且每小时都在变化;而内容库只有在新增内容时才会改变。要回答这类查询的系统,必须持续获取趋势信号,并在请求时将其与内容库进行比对。

正是这一区别决定了核心的架构决策:内容理解与趋势获取作为两条独立的流水线,按各自的节奏运行,只有当用户选择某个 QuickSearch 操作时才会被合并。

借助 QuickSearch,团队可以以另一种方式与视频库交互——由系统主动推荐,而非被动等待。它提供四种操作,每种操作都有自己的检索策略、编辑标准和排序模型。接下来的操作演示描述了每种操作是如何实现的。而这里的列表则介绍它们各自的作用。

除此之外,Social Department 还保留了自然语言搜索,用于覆盖这些操作未涉及的一切内容;以及基于图像的搜索——用一张参考图片即可在整个库中返回视觉上相似的内容。

下图展示了该架构三个部分之间的连接方式。内容理解流水线让视频库变得可检索,趋势获取通道则引入外部趋势信号。QuickSearch 任务层在请求时把两者连接起来。

图 1:高层架构

该系统由三个部分组成,各自按独立的调度运行。其中两个负责获取并结构化信息。第三个则把用户的单次选择解析为跨越前两者的多阶段检索操作。

这条流水线让视频库变得可读。它在数据摄入时异步运行。一个 AWS Step Functions 状态机端到端编排约二十几个 Python AWS Lambda 函数。以下步骤追踪一个视频从上传到完成索引片段的完整过程。

这条流水线端到端都是事件驱动的。每个 Lambda 任务都会以指数退避方式重试。场景描述阶段还额外增加了带完全抖动的专用限流重试。非关键的富集步骤会优雅降级,而不是让整个运行失败;而硬失败则会路由到错误处理器,由它记录原因并将该资产报告为失败。

Marengo 的同区域暂存。Marengo 仅在特定的一组 AWS 区域中可用。Amazon Bedrock 中的异步调用会从与端点位于同一区域的 Amazon S3 存储桶读取输入,并将输出写入该桶。整条流水线正是围绕这些参数构建的。它会在调用 Marengo 之前,先把每个视频暂存到 US East(N. Virginia)的一个专用存储桶中,然后再从同一个桶读回嵌入输出。其余基础设施仍留在其所属区域,而每个嵌入作业都与模型一同运行。关于各区域的模型可用性,请参见 Amazon Bedrock 中“按 AWS 区域划分的模型支持”。

这条流水线持续运行,且独立于任何内容库,正是它让 Trend Align 成为可能。

Quotable Quotes 则依赖第二种更窄的输入:一个外部对白热度信号,仅用于对已经流传开的台词加权。它在设计上就是可选的,因此即使该信号不可用,某句台词也只是得不到加权,而不会导致任务失败。

这一层正是把单次选择转化为检索策略的地方。它异步执行每个动作,并返回一组经过排序的片段。

有三项选择让融入文化信息的检索变得可行。

内容库只在新内容加入时才发生变化。而趋势每天都在变,处于早期阶段的趋势变化更快。若把两者耦合,要么得用最新信号重新处理内容——这在算力上根本不可行——要么只能输出过时的趋势判断。让它们作为彼此独立的流水线、仅在请求时汇合,意味着信号始终保持最新,而内容库再也不需要被改动。一部多年前就已播完的剧集,可以重新进入当下的讨论。

朴素的设计会把一个大模型放在整个库的前面。QuickSearch 则不同:它先用低成本、高召回率的向量与关键词检索构建候选池,然后只把模型预算花在对一组数量受限的真实候选做排序上。这个数量上限是针对排序模型的输出 Token 限制来设定的,因为排序模型如果在回答中途被截断,就会悄无声息地丢掉结果。这也让 Amazon Bedrock 的单次查询成本保持可预测:一次 embedding 调用、一次 rerank 调用,以及数量固定且不多的 Claude 调用,与库的规模无关。

结果会被缓存 30 天。若不加以干预,对某个 action 的判定标准或模型所做的修改,在缓存过期之前对用户都是不可见的。每个 action 都带有一个流水线版本号,凡是改变其行为的改动都会在同一个变更中把这个版本号递增。版本号一递增,此前的缓存结果立即失效,于是编辑判断的调整就成为对产品的一次带版本号的变更,而不必在每次查询时重新争论。

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(How Social Department built culture-aware video discovery)

This post is co-written by Mariah Miller and Sirajus Salekin from AWS and Nicholas Chase from Social Department.

Entertainment marketing teams manage tens of thousands of video assets across studios, networks, and streaming applications. Most campaigns begin with the same video discovery problem: finding the one moment in the footage that will resonate with an audience. When a brief lands, the clock starts, and what they need is almost never something a database can describe.

Semantic search, on its own, has a ceiling: It answers the question you can articulate. Ask it for a character holding a coffee mug and it will find every coffee mug in a thousand episodes. The queries that actually drive entertainment campaigns look nothing like that. They are questions of judgment: which moment in a show will travel, which line is quotable, and which single frame will work as a meme.

Those questions can’t be typed into a search box. Answering them requires two things the search index doesn’t have: an editorial definition of what “quotable” or “memeable” means for a given library, and a continuously refreshed view of what is moving outside the library. Historically, answering them meant a person with deep knowledge of the show watching footage for a week, and that person doesn’t scale across a catalog.

In this post, we describe how Social Department, the AI operating system for entertainment marketing, set out to encode those editorial questions as executable retrieval pipelines. The result is QuickSearch, a set of single-action queries that return the clips a marketer is looking for without requiring them to describe it: Shareable Scenes, Quotable Quotes, Memeable Moments, and Trend Align.

The models in this solution run on Amazon Bedrock, so customer data stays within the AWS data boundary. It isn’t shared with model providers and isn’t used to train the underlying models.

Social Department’s approach to responsible AI is that AI proposes and people decide. In production, code checks generated output against the source material (for example, a quote is discarded if it isn’t in the episode transcript). Amazon Bedrock Guardrails helps screen customer search text for prompt attacks, and a person on the customer’s team approves every post before it publishes.

Social Department owns the pipeline architecture, the retrieval system, and the editorial criteria behind each action. TrackIt, an AWS Premier Tier Services Partner, implemented and delivered against that design. TwelveLabs’ Marengo 3.0 and Pegasus 1.2 models on Amazon Bedrock provide the scene-level video reasoning the pipeline builds on. AWS supplies the infrastructure, the model access including Anthropic’s Claude models, and Amazon OpenSearch Service, using its serverless option, Amazon OpenSearch Serverless, for retrieval.

A note on scope: This post describes the AWS portions of the system: the ingest and indexing pipeline, the retrieval path, and the services that run them. The components Social Department owns, including its media and data backbone, content analysis and discovery engine, editorial criteria, and evaluation program, are described only to the depth needed to follow the architecture. The named services that follow are the ones this post is about, not an inventory of Social Department’s full system.

About Social Department

Social Department is the AI operating system for entertainment marketing, serving major studios, networks, streamers, and FAST applications. It automates campaign strategy, asset creation, publishing, and performance optimization across a client’s full content library. Most of those functions depend on locating the right moment inside the footage first, and semantic clip discovery has been core to the product from the beginning.

Why semantic search alone was insufficient

Entertainment marketing operates under two constraints that a general-purpose search layer can’t satisfy.

The valuable queries are the ones nobody can phrase

A marketer can describe an object, a location, or a line of dialogue, but not what will travel. That judgment, this moment lands and that one doesn’t, is the actual skill in the job. It was never automatable because it depends on knowing how an audience behaves rather than what a video contains. A search layer that only understands content requires a human to supply the intent, and supplying that intent well requires having already watched the show.

Trend signal lives outside the library and moves on a different clock

No amount of video understanding tells you what audiences are talking about this morning. That information is created outside the library and changes hourly, while a library changes only when new content is added. A system that answers these queries has to acquire trend signal continuously and reconcile it against the library at request time.

That distinction drove the core architectural decision: content understanding and trend acquisition run as independent pipelines on their own schedules and are joined only when a user selects a QuickSearch action.

Solution overview

With QuickSearch, teams can interact with a video library in a different way, one where the system proposes rather than waits. It provides four actions, each with its own retrieval strategy, editorial criteria, and ranking model. The walkthrough that follows describes how each is implemented. The list here introduces what they do.

  1. Shareable Scenes surfaces substantial, self-contained scenes such as extended confrontations, climaxes, and set pieces, biased toward developed sequences rather than quick reactions.
  2. Quotable Quotes finds punchy, quotable dialogue capped at short clip lengths, weighting up lines that have demonstrably circulated where that signal is available for the title.
  3. Memeable Moments identifies individual frames that will function as memes and generates a caption grounded in the actual frame. A dedicated extraction and scoring step chooses the frame rather than taking a thumbnail.
  4. Trend Align takes what is trending right now and finds the moments in the library that legitimately connect to it, so a title that finished airing years ago can re-enter a live conversation.

Alongside these, Social Department keeps natural-language search for everything the actions don’t cover, and image-based search, where a reference image returns visually similar content across the library.

The following diagram shows how the three parts of the architecture connect. The content understanding pipeline makes the library searchable, and the trend acquisition lanes pull in external trend signal. The QuickSearch job layer joins the two at request time.

Architecture connecting the content understanding pipeline, trend acquisition lanes, and the QuickSearch job layer

Figure 1: High-level architecture

Architecture walkthrough

The system has three parts that run on independent schedules. Two acquire and structure information. The third resolves a user’s single selection into a multi-stage retrieval operation across both.

Content understanding pipeline

This pipeline makes the library legible. It works asynchronously on ingest. An AWS Step Functions state machine orchestrates roughly two dozen Python AWS Lambda functions end to end. The following steps trace one video from upload to indexed clips.

  1. Upload and trigger – Video lands in a per-team Amazon Simple Storage Service (Amazon S3) bucket. The S3 Object Created event, routed through Amazon EventBridge, starts the Step Functions execution.
  2. Metadata – The first Lambda function resolves title, team, episode, and public identifiers against Social Department’s media and data backbone, writes video metadata to Amazon DynamoDB, and reports the asset as processing.
  3. Transcode – AWS Elemental MediaConvert transcodes the asset to 480p H.264 with adaptive deinterlacing, using a Step Functions task token so the state machine waits on the job rather than polling.
  4. Shot detection – Amazon Rekognition performs asynchronous segment detection with an Amazon Simple Notification Service (Amazon SNS) callback. A second Lambda function merges and splits the raw shots into content-aware ranges of roughly 10–20 seconds. Those ranges are the unit the rest of the pipeline fans out over. Within a range, Social Department segments embeddings at a fixed interval, a design choice tuned to the pacing of episodic television. The indexed clip is short and fixed-length, but the range it sits in follows the cut.
  5. Embedding – Ranges are processed in a Step Functions Distributed Map. Each call passes a start and end time to TwelveLabs Marengo 3.0 on Amazon Bedrock through asynchronous invocation. The call produces multimodal embeddings across visual, audio, and transcript dimensions, with no file splitting needed on this path.
  6. Scene description – TwelveLabs Pegasus 1.2 on Amazon Bedrock runs per segment in a parallel Distributed Map, producing scene-level narrative descriptions, emotions, themes, dialogue type, and narrative position, which form the text layer the ranking models later reason over. Pegasus takes pre-cut files rather than time ranges, so a separate Lambda function cuts each range first, with a few seconds of padding on either side to preserve scene transitions.
  7. Transcription and enrichment – In parallel with embedding, the pipeline runs speech-to-text with automatic fallback, and Amazon Rekognition for on-screen text, visual labels, and face collections used in character grounding. Anthropic’s Claude models on Amazon Bedrock extract the show’s character map and generate show-level concepts and character roles.
  8. Indexing – Amazon Bedrock writes embedding output to Amazon S3. A final Distributed Map then indexes embeddings, transcript, on-screen text, scene description fields, and labels into Amazon OpenSearch Serverless.

The pipeline is event-driven end to end. Every Lambda task retries with exponential backoff. The scene-description stage adds a dedicated throttling retry with full jitter. Non-critical enrichment steps degrade gracefully rather than failing the run, and a hard failure routes to an error handler that records the reason and reports the asset as failed.

Same-Region staging for Marengo. Marengo is available in a specific set of AWS Regions. Asynchronous invocation in Amazon Bedrock reads input from and writes output to an Amazon S3 bucket in the same Region as the endpoint. The pipeline is built around those parameters. It stages each video into a dedicated bucket in US East (N. Virginia) before invoking Marengo, then reads the embedding output back from the same bucket. The rest of the infrastructure stays in its home Region while each embedding job runs alongside the model. For model availability by Region, see Model support by AWS Region in Amazon Bedrock.

Trend acquisition

This pipeline operates continuously and independently of any library, and it is what makes Trend Align possible.

  1. Two lanes, one feed – Trend data is acquired through two lanes with deliberately different mechanisms. The daily lane uses an Amazon EventBridge cron, one scheduled fetch per region per day, timed to each region’s local morning rather than a fixed UTC hour. The heat lane (Heat Seeker) reads early-stage, momentum-ordered trends, and is fetched only when a user explicitly asks for it, reusing what is already stored if it is recent enough. The split is a cost decision. A cron pays for every region on every tick whether or not anyone is working, while a demand-shaped fetch behind a TTL costs nothing on a quiet day.
  2. Regional feeds are separate populations – Trends are acquired per geographic region, currently four feeds: worldwide, the United States, the United Kingdom, and Australia. A region isn’t a filter over a global list. Each is discovered separately on its own schedule, so the same day’s trends genuinely differ between them.
  3. Interpretation and the editorial gate – Raw trend data isn’t shown to a customer. Anthropic’s Claude Haiku on Amazon Bedrock normalizes each trend onto Social Department’s own category schema, standardizes its description, deduplicates near-matches, and caps how many survive per category. In the same pass, an editorial gate drops trends whose primary subject is death, violence, self-harm, natural disaster, politics, protest, crime, or a public-health crisis. A marketing customer isn’t offered these, regardless of how large the trend is. Rejected trends are remembered so each fetch doesn’t re-screen them.
  4. Storage as first-class rows – Interpreted trends are upserted by stable fingerprint, so the same trend day-over-day reuses its row and its cached derivations. Trends that stop appearing age out and are eventually retired. Refresh runs take a claim so two cannot overlap.

Quotable Quotes draws on a second, narrower input: an external dialogue popularity signal, used strictly to weight lines that have already circulated. It’s optional by design, so if the signal is unavailable a line gets no boost rather than failing the job.

The QuickSearch job layer

This layer is where a single selection becomes a retrieval strategy. It executes each action asynchronously and returns a ranked set of clips.

  1. Request path – The web application is a React single-page application served from Amazon S3 through Amazon CloudFront. Its API is a Fastify service running on AWS Lambda behind an Amazon API Gateway HTTP API, with Amazon Aurora PostgreSQL-Compatible Edition for relational data.
  2. Asynchronous execution – Selecting a QuickSearch action writes a job record to Amazon DynamoDB and enqueues an Amazon Simple Queue Service (Amazon SQS) message. A worker Lambda function picks it up and runs the pipeline for that action. Transient failures retry through Amazon SQS, and errors classified as non-retriable go straight to a dead-letter queue.
  3. Query formulation – Each action expands into a set of retrieval queries shaped by its editorial criteria. For Trend Align, this is where most of the proprietary work sits: A trend arrives as a cultural topic, and what a video index can match is what appears on screen and in the audio. Bridging those two is a translation problem, which Claude models on Amazon Bedrock handle. The output is a set of queries the index can act on.
  4. Multi-channel retrieval – Each query goes to the search Lambda function, which handles hybrid retrieval. Marengo embeds the query once, as text, an image, or a composed text-plus-image query. The embedded query then executes across six k-NN channels (visual, audio, transcript, on-screen text, scene description, and labels) and seven BM25 text channels. Per-modality weights come from the query itself. Query-type routing adjusts channel boosts so dialogue queries lean on transcript and emotion channels while visual queries suppress them.
  5. Fusion and reranking – The channels are combined with Reciprocal Rank Fusion, which uses rank positions rather than raw scores and so avoids the scale mismatch between k-NN and BM25 similarity. The top candidates then pass through the Amazon Rerank model, a cross-encoder called through the Rerank API in Amazon Bedrock, for a final text-to-text relevance pass, followed by a Claude validation step over the top results.
  6. Aggregation – The index stores short fixed-length clips. Before results are returned, adjacent clips with high visual-embedding similarity are merged into coherent moments, so a query for a scene returns one 20–60 second result rather than five fragments of the same scene.
  7. Editorial ranking – A Claude model on Amazon Bedrock ranks the deduplicated, enriched candidates against the action’s editorial criteria. The pool is capped so the model’s output can’t truncate. Model choice is per action: Claude Sonnet where cultural recall changes the answer, Claude Haiku where the task is bounded and cost-sensitive.
  8. Frame extraction – For Memeable Moments, a dedicated Lambda function extracts several candidate frames from the winning clip and scores them for face quality using Amazon Rekognition and for image sharpness. It then re-extracts the winner at full resolution, trims letterboxing, and returns a signed Amazon CloudFront URL.
  9. Caching – Completed results are stored in Amazon DynamoDB and reused for 30 days. The cache key includes a per-action pipeline version and, for Trend Align, a hash of the active trend set, so a criteria change, a model upgrade, or a refreshed trend catalog invalidates cleanly rather than serving a stale answer.

Design decisions

Three choices made culturally informed retrieval tractable.

Separating trend acquisition from content understanding

A library changes when new content is added. Trends change daily, and early-stage trends faster than that. Coupling the two would mean either reprocessing content against fresh signal, which is computationally untenable, or serving stale trend judgment. Keeping them as independent pipelines joined at request time means signal stays current without the library ever being touched again. A title that finished airing years ago can re-enter a live conversation.

Splitting each action into cheap retrieval and expensive judgment

The naive design puts one large model in front of the whole library. QuickSearch instead uses inexpensive, high-recall vector and keyword retrieval to build a candidate pool, then spends model budget only on ranking a capped set of real candidates. The caps are sized against the ranking model’s output token limit, because a ranker that truncates mid-response silently drops results. This also keeps the per-query cost of Amazon Bedrock predictable: one embedding call, one rerank call, and a small fixed number of Claude invocations, regardless of library size.

Making cache invalidation an explicit editorial act

Results are cached for 30 days. Without intervention, a change to an action’s criteria or model would stay invisible to users until the cache expired. Each action carries a pipeline version that is bumped in the same change that alters its behavior. That bump invalidates prior results immediately, so editorial judgment is refined as a versioned change to the product rather than relitigated on every query.

Results

Social Department reports the following outcomes from customer use of QuickSearch. These are the company’s own observations rather than a controlled benchmark.

  1. Queries that were previously impossible are now a single selection – Identifying the moments a fandom has canonized, or the frames that will function as memes for a specific audience, previously required a person with deep familiarity with the title watching footage, and it didn’t scale across a catalog.
  2. Discovery no longer depends on manual scrubbing – According to Social Department, campaign discovery work that used to take 3–4 days now takes about 10 minutes with QuickSearch. The time a marketer spent scrubbing through episodes for the right moment goes back to higher-value campaign work.
  3. Marketers reach candidate clips without composing a query – That removes the requirement that the person searching already knows the library well enough to describe what they want, the constraint that made discovery depend on the few people who had watched the show.
  4. Archived catalog becomes addressable – Content that was effectively unreachable, because it was finished, archived, and invisible to metadata search, becomes usable for campaign work, and titles that stopped airing years ago can re-enter live cultural conversations.
  5. Industry recognition – Social Department was showcased as a top startup at NAB 2026 and featured in the Entertainment Loft.

“The hard part here was never the search. It was working out what to search for: how you take something a great marketer does on instinct, about which moment lands and why, and turn it into criteria a system can execute a thousand times. We spent a long time on that. Working with TrackIt on a compressed timeline, we went from spec to a live demo at NAB. TwelveLabs gives us the intelligence to understand what’s inside the footage and AWS gives us the infrastructure to do it at library scale. Our job is to know what to point it at.”

— Jonathan Verk, Founder and CEO, Social Department

“Everyone in media talks about adding metadata to their libraries but tagging content doesn’t pay for itself. What I like about this project is that it closes the loop. It takes shows sitting in an archive and makes them useful for real campaigns, at a cost that makes sense to run every day. That’s the part the industry has been missing.”

— Chris Marchitelli, Vice President, TrackIt

What we learned

The following four lessons from the project stood out.

The hard problem was expressing judgment, not retrieving content

Retrieval is largely solved. What isn’t solved is specifying intent precisely enough for retrieval to be useful without an expert marketer in the loop. Most of the engineering effort went into defining what qualifies as a shareable scene or a memeable frame, tightly enough to execute repeatedly across libraries with different tones and audiences. That definition is the output of a standing evaluation program of numbered iterations, each measured against per-show ground-truth sets and a tracked baseline, that tunes the criteria against results rather than intuition.

A multi-model architecture on Amazon Bedrock

QuickSearch uses a multi-model architecture, with each model assigned to the work it fits best. Claude Haiku handles bounded, cost-sensitive output, and Claude Sonnet handles tasks where cultural recall changes the answer. TwelveLabs Marengo produces multimodal embeddings, TwelveLabs Pegasus provides scene-level video reasoning, and the Amazon Rerank model handles reranking. Amazon Bedrock made this architecture practical to build and run. Models from TwelveLabs, Anthropic, and Amazon are available through one API, so Social Department can change the model behind a task when measured quality moves, without building a new integration. Asynchronous invocation runs long video embedding jobs at library scale, and the Rerank API adds a relevance pass without separately hosted infrastructure. Amazon Bedrock Guardrails helps screen search input in the same service, and customer data stays within the AWS data boundary. Because Amazon Bedrock is fully managed, the team operates no model-serving infrastructure of its own.

Signal that decays needs its own architecture

Treating trend data as another input to the content pipeline would have failed, because the two have incompatible lifecycles. Recognizing early that library content and live culture are separate systems, with separate refresh rates, storage, and failure modes, was the decision the rest of the design depended on.

Clear layer boundaries made a multi-party project work

Social Department owned the pipeline architecture, retrieval system, and editorial criteria. TrackIt built and delivered against that architecture, and TwelveLabs supplied the video understanding models. AWS, with the AWS Generative AI Innovation Center, provided the infrastructure, model access, and solution architecture for the AWS components. Because each party had a clearly defined scope, the 8-week production timeline held.

Conclusion

We described how Social Department, working with TrackIt, TwelveLabs, and AWS, built culture-aware video discovery for entertainment marketing using its own pipeline and retrieval architecture. Several AWS components make up the layer this post describes. TwelveLabs Marengo handles multimodal embedding and TwelveLabs Pegasus handles scene-level video reasoning, both on Amazon Bedrock. Anthropic’s Claude models on Amazon Bedrock handle interpretation and editorial ranking, and Amazon OpenSearch Serverless handles hybrid retrieval. AWS Step Functions, AWS Lambda, AWS Elemental MediaConvert, Amazon Rekognition, Amazon DynamoDB, and Amazon SQS run the pipelines around them. The result is a set of single-action queries that return the clips audiences already care about, the lines they already quote, the frames that will work as memes, and the moments that connect to whatever is trending right now.

To get started with multimodal video understanding on AWS, explore Amazon Bedrock and Amazon OpenSearch Service. To learn more about the TwelveLabs models available on Amazon Bedrock, visit the TwelveLabs page on AWS Marketplace. To explore how the AWS Generative AI Innovation Center can help your organization, visit the AWS Generative AI Innovation Center page.

Acknowledgements: The authors would like to thank Katie Kenney, Pranay Duhan, Xiaoye Qian, Rinne Han, and Dahler Battle for their work on this solution. We would also like to thank the TrackIt and TwelveLabs teams for their support. 


About the authors

Mariah Miller

Mariah Miller

Mariah is a Senior Generative AI Strategist and industry leader for media, entertainment, games, and sports in AWS Forward Deployed Engineering, a role she also held at the AWS Generative AI Innovation Center. She leads customer missions that build production AI systems with enterprise teams across industries, including financial and legal services, and develops reusable solution assets such as multimodal video search, agentic commerce, and voice AI agents. She also built the Innovation Center’s partner program. Outside of work, she likes to spend time outdoors.

Sirajus Salekin

Sirajus Salekin

Sirajus Salekin is an Applied Scientist at AWS Forward Deployed Engineering (formerly the AWS Generative AI Innovation Center). He works with customers and AWS Partners to design and build scalable generative AI solutions, from early experiments through production. His work spans generative AI, multimodal ML, NLP, and video search, combining applied research with delivery to make sure every solution is tied to real business outcomes.

Nicholas Chase

Nicholas Chase

Nicholas is a Lead Software Engineer at Social Department, the AI operating system for entertainment marketing, where he leads system architecture and the video search stack. Chase drove the retrieval requirements and search quality work described in this post.

正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 AWS Machine Learning(How Social Department built culture-aware video discovery),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗