11-19 更新 实时
首页/ AI 大模型/ 正文

用 Scikit-LLM 集成 LLM 翻译流水线

AI 大模型 来源:KDnuggets 2026-11-10 23:00 0 阅读 机器翻译
用 Scikit-LLM 集成 LLM 翻译流水线

作者:Iván Palomares Carrascosa,KDnuggets 技术内容专家,2026年11月10日,语言模型栏目

文本分类有时需要处理用不同语言写成的原始文本,同时还面临一个额外挑战:必须先把所有输入数据归一化为同一种语言,再交给下游分类模型。所幸,Scikit-LLM 不仅能帮助构建由大语言模型(LLM)驱动的流水线,用于零样本和少样本分类等任务,它还提供了自定义类,可将预训练模型适配到翻译等其他专门的语言任务上。

本文将展示如何为多语言文本分类构建一个贯穿始终的 scikit-learn 流水线,它由两个主要阶段组成,二者都由 Scikit-LLM 封装的模型驱动:翻译和零样本分类。我将教你构建的这个流水线有多项优势:

借助一个为执行零样本分类而封装的预训练 LLM,它以语义推理取代了传统文本分类器僵硬的关键词匹配

输入文本被统一为同一种语言,从而能够用单个下游模型实现有效分类

使用 Scikit-LLM 的零样本分类可解决冷启动数据问题,只需类别名称(类标签)以及少量相关训练样本就能开始工作:完全不需要额外的模型训练!

构建多语言翻译与零样本分类流水线

本示例可以完全在 Google Colab notebook 中运行。它使用 Groq 的 LLM API,完全依赖免费模型。前提是:前往 Groq 网站注册,并生成你自己的 API key。生成后,请仔细复制你的 API key(它只会显示一次),并在 Google Colab 中把它粘贴为一个新的“Secret”变量——该功能位于 Colab 左侧的钥匙图标下。在这里粘贴你复制的 API key,并将该 secret 变量命名为 GROQ_API_KEY。

之后,我们通过安装 Scikit-LLM 来开始我们的 notebook:

!pip install scikit-llm -q

现在,让我们开始编写 Python 代码。第一步是配置 Scikit-LLM 所处的环境,将我们的请求改为经由 Groq 的 API 路由,而不是默认的 OpenAI API。这样做,我们就能利用 Groq 免费、易用的预训练模型,避免按 Token 计费。

from google.colab import userdata from skllm.config import SKLLMConfig # 获取 API key,并路由到免费层级的 OpenAI 兼容端点(例如 Groq) groq_key = userdata.get('GROQ_API_KEY') SKLLMConfig.set_gpt_url("https://api.groq.com/openai/v1") SKLLMConfig.set_openai_key(groq_key)

接下来,我们定义“训练”数据。由于我们打算使用零样本 LLM 分类器——而不是先用向量化器把文本映射为数值、再使用传统机器学习分类器——我们的优势在于不需要大量且类别均衡的训练样本。我们只需要定义模型必须学会识别并用于预测的那组唯一分类标签,再加上几个占位数据样本,以满足 Scikit-LLM 的 fit() 方法的最低要求——该方法模仿了其在 scikit-learn 中的对应方法:

# 1. 多语言训练数据 # 加入少量混合语言、多个主题的标注样本 X_multilingual_texts = [ # Energy "La technologie des panneaux solaires a vu son efficacité augmenter de 15%.", # French (Solar) "Neue Batteriespeicher für Windenergie werden immer wirtschaftlicher.", # German (Wind/Battery) # Finance "Los mercados financieros sufrieron una fuerte caída hoy tras la subida de tipos.", # Spanish (Markets) "Les actions technologiques ont rebondi grâce à de solides rapports sur les bénéfices.", # French (Stocks/Earnings) # Healthcare "Die klinische Studie für das neue Medikament reduzierte die Schwellung um 40%.", # German (Medication/Trial) "Un ensayo reciente demostró que la vacuna es muy eficaz para prevenir infecciones.", # Spanish (Vaccine) ] y_labels = ["Energy", "Energy", "Finance", "Finance", "Healthcare", "Healthcare"]

下一步是核心环节:我们定义一条两阶段流水线,先做翻译,再做零样本分类。正是在这里,我们把 Scikit-LLM 将预训练 LLM 适配到特定任务的能力,整合进一个更宽泛的、基于 scikit-learn 的机器学习流水线中。

下面的代码使用了 Scikit-LLM 中两个面向特定任务的类:GPTTranslator 用于将一个基础 LLM 适配为翻译成通用语言,ZeroShotGPTClassifier 则用于将另一个 LLM 适配到我们流水线中的后续任务,也就是对翻译后的文本进行分类,而无需重新训练模型。请注意,两个阶段都选用同一个基础 Groq 模型——"custom_url::llama-3.1-8b-instant"——完全没有问题:我们只是用不同的类把两个模型实例适配到不同的目标任务上。

from sklearn.pipeline import Pipeline from skllm.models.gpt.text2text.translation import GPTTranslator from skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier # 2. 翻译与零样本分类流水线 # 一个基于 scikit-learn 的流水线,其中包含两个由 Scikit-LLM 封装的模型 text_pipeline = Pipeline([ ('translator', GPTTranslator( model="custom_url::openai/gpt-oss-20b", output_language="English" )), ('classifier', ZeroShotGPTClassifier( model="custom_url::openai/gpt-oss-20b" )) ]) # 3. 拟合分类器 # ZeroShotGPTClassifier 只需看到唯一的标签即可理解各个类别。 # 仅为满足 scikit-learn API 的要求,传入一个最小化的占位列表作为 X。 y_labels_unique = ["Energy", "Finance", "Healthcare"] X_placeholder = [""] * len(y_labels_unique) text_pipeline.fit(X_placeholder, y_labels_unique)

在向刚刚搭建好的流水线传入若干新样本、做出最后几次预测之前,我们先看看两个模型中第一个模型为训练样本生成的翻译,以确认它们是正确的:

# 访问 'translator' 步骤以转换训练数据 train_translations = text_pipeline.named_steps['translator'].transform(X_multilingual_texts) print("--- TRAINING DATA TRANSLATIONS ---") for label, original, translated in zip(y_labels, X_multilingual_texts, train_translations): print(f"[{label.upper()}]\nOriginal: {original}\nTranslated: {translated}\n")

--- 训练数据翻译 --- [能源] 原文:La technologie des panneaux solaires a vu son efficacité augmenter de 15%. 译文:太阳能板技术的效率提升了15%。 [能源] 原文:Neue Batteriespeicher für Windenergie werden immer wirtschaftlicher. 译文:用于风能的新型电池存储正变得越来越经济。 [金融] 原文:Los mercados financieros sufrieron una fuerte caída hoy tras la subida de tipos. 译文:在加息之后,金融市场今天遭遇了大幅下跌。 [金融] 原文:Les actions technologiques ont rebondi grâce à de solides rapports sur les bénéfices. 译文:科技股因强劲的财报而反弹。 [医疗] 原文:Die klinische Studie für das neue Medikament reduzierte die Schwellung um 40%. 译文:这款新药的临床研究使肿胀减轻了40%。 [医疗] 原文:Un ensayo reciente demostró que la vacuna es muy eficaz para prevenir infecciones. 译文:最近的一项研究显示,该疫苗在预防感染方面非常有效。

既然这些翻译看起来相当不错,我们现在可以迈出最后一步,想出一些多语言的测试示例,然后通过预先翻译将它们传给流水线进行分类:

# 4. 在未见数据上进行预测 X_new_multilingual = [ "Un nuovo vaccino antinfluenzale ha mostrato un'efficacia dell'85% negli studi clinici, promettendo difese immunitarie più forti.", # 意大利语(医疗健康) "I ricercatori hanno sviluppato una batteria agli ioni di sodio che immagazzina l'energia eolica in eccesso in modo molto economico.", # 意大利语(能源) "As ações de tecnologia se recuperaram no final do pregão, impulsionadas por fortes relatórios de lucros corporativos." # 葡萄牙语(金融) ] predictions = text_pipeline.predict(X_new_multilingual) new_translations = text_pipeline.named_steps['translator'].transform(X_new_multilingual) print("\n--- 未见数据预测 ---") for text, translation, prediction in zip(X_new_multilingual, new_translations, predictions): print(f"原始文档: '{text}'") print(f"中间翻译: '{translation}'") print(f"最终预测类别: {prediction}\n" + "-" * 70)

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Integrating LLM Translation into Your Scikit-Learn Pipelines with Scikit-LLM)

By Iván Palomares Carrascosa, KDnuggets Technical Content Specialist on November 10, 2026 in Language Models

Integrating LLM Translation into Your Scikit-Learn Pipelines with Scikit-LLM

Text classification sometimes requires dealing with raw texts written in different languages, with the added challenge of normalizing all incoming data into a unified language before providing it to the downstream classification model. Thankfully, Scikit-LLM not only facilitates the construction of pipelines driven by large language models (LLMs) for tasks like zero-shot and few-shot classification. It also provides custom classes to adapt pre-trained models for other specialized language tasks like translation.

This article shows how to build an overarching scikit-learn pipeline for multilingual text classification, consisting of two major stages, both guided by Scikit-LLM-wrapped models: translation and zero-shot classification. The resulting pipeline I'll teach you to build has multiple advantages:

It replaces the rigid keyword matching of traditional text classifiers with semantic reasoning, thanks to a pre-trained LLM wrapped for zero-shot classification

Incoming texts are unified into a common language to enable effective classification with a single downstream model

Zero-shot classification with Scikit-LLM addresses the cold-start data issue, needing only category names (class labels) along with a few associated training examples to start doing its job: no extra model training needed at all!

Building a Multilingual Translation and Zero-Shot Classification Pipeline

This example can be run entirely in a Google Colab notebook. It uses Groq's LLM API to rely entirely on free models. As a prerequisite, go to the Groq website, register, and generate your own API key. Once it is generated, carefully copy your API key (it is shown only once) and paste it as a new "Secret" variable in Google Colab, found under the key icon on the left-hand side of Colab. Here, you'll paste your copied API key and name the secret variable GROQ_API_KEY.

After that, we start our notebook by installing Scikit-LLM:

!pip install scikit-llm -q

Now, let's start writing our Python code. The first step is to configure the environment surrounding Scikit-LLM, routing our requests through Groq's API instead of the default OpenAI API. By doing this, we leverage Groq's free, user-friendly pre-trained models, avoiding per-token costs.

from google.colab import userdata from skllm.config import SKLLMConfig # Retrieving API key and route to a free-tier OpenAI-compatible endpoint (e.g. Groq) groq_key = userdata.get('GROQ_API_KEY') SKLLMConfig.set_gpt_url("https://api.groq.com/openai/v1") SKLLMConfig.set_openai_key(groq_key)

Next, we define the "training" data. Since we are going to use a zero-shot LLM classifier — instead of a traditional machine learning classifier preceded by a vectorizer that maps text numerically — we have the advantage of not needing a large, balanced set of training examples. All we need is to define the unique set of classification labels that the model must learn to recognize and use for predictions, as well as a few placeholder data examples that fill the minimal requirements of Scikit-LLM's fit() method, which mimics its scikit-learn counterpart:

# 1. MULTILINGUAL TRAINING DATA # Adding a few labeled examples in mixed languages and multiple topics X_multilingual_texts = [ # Energy "La technologie des panneaux solaires a vu son efficacité augmenter de 15%.", # French (Solar) "Neue Batteriespeicher für Windenergie werden immer wirtschaftlicher.", # German (Wind/Battery) # Finance "Los mercados financieros sufrieron una fuerte caída hoy tras la subida de tipos.", # Spanish (Markets) "Les actions technologiques ont rebondi grâce à de solides rapports sur les bénéfices.", # French (Stocks/Earnings) # Healthcare "Die klinische Studie für das neue Medikament reduzierte die Schwellung um 40%.", # German (Medication/Trial) "Un ensayo reciente demostró que la vacuna es muy eficaz para prevenir infecciones.", # Spanish (Vaccine) ] y_labels = ["Energy", "Energy", "Finance", "Finance", "Healthcare", "Healthcare"]

The next step is the central one: we define our two-stage pipeline for translation followed by zero-shot classification. This is where we integrate Scikit-LLM's capabilities for adapting pre-trained LLMs to specific tasks into a broader, scikit-learn-based machine learning pipeline.

The code below uses two task-specific classes from Scikit-LLM: GPTTranslator to adapt a base LLM for translation into a common language, and ZeroShotGPTClassifier to adapt another LLM for the subsequent task in our pipeline, i.e. classification of the translated text without the need to retrain the model. Notice below that it is perfectly fine to choose the same base Groq model — "custom_url::llama-3.1-8b-instant" — for both stages: we just use different classes to adapt the two model instances to different target tasks.

from sklearn.pipeline import Pipeline from skllm.models.gpt.text2text.translation import GPTTranslator from skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier # 2. THE TRANSLATION AND ZERO-SHOT CLASSIFICATION PIPELINE # A scikit-learn-based pipeline that contains two Scikit-LLM-wrapped models text_pipeline = Pipeline([ ('translator', GPTTranslator( model="custom_url::openai/gpt-oss-20b", output_language="English" )), ('classifier', ZeroShotGPTClassifier( model="custom_url::openai/gpt-oss-20b" )) ]) # 3. FITTING THE CLASSIFIER # ZeroShotGPTClassifier only needs to see the unique labels to understand the categories. # Passing a minimal placeholder list for X only to satisfy the scikit-learn API requirements. y_labels_unique = ["Energy", "Finance", "Healthcare"] X_placeholder = [""] * len(y_labels_unique) text_pipeline.fit(X_placeholder, y_labels_unique)

Before making a few final predictions by passing fresh examples to our newly built pipeline, we take a look at the translations produced by the first of the two models for the training examples to verify that they are correct:

# Accessing the 'translator' step to transform the training data train_translations = text_pipeline.named_steps['translator'].transform(X_multilingual_texts) print("--- TRAINING DATA TRANSLATIONS ---") for label, original, translated in zip(y_labels, X_multilingual_texts, train_translations): print(f"[{label.upper()}]\nOriginal: {original}\nTranslated: {translated}\n")

--- TRAINING DATA TRANSLATIONS --- [ENERGY] Original: La technologie des panneaux solaires a vu son efficacité augmenter de 15%. Translated: The technology of solar panels has seen its efficiency increase by 15%. [ENERGY] Original: Neue Batteriespeicher für Windenergie werden immer wirtschaftlicher. Translated: New battery storage for wind energy is becoming increasingly economical. [FINANCE] Original: Los mercados financieros sufrieron una fuerte caída hoy tras la subida de tipos. Translated: Financial markets suffered a sharp drop today after the rate hike. [FINANCE] Original: Les actions technologiques ont rebondi grâce à de solides rapports sur les bénéfices. Translated: Tech stocks rebounded thanks to solid earnings reports. [HEALTHCARE] Original: Die klinische Studie für das neue Medikament reduzierte die Schwellung um 40%. Translated: The clinical study for the new drug reduced swelling by 40%. [HEALTHCARE] Original: Un ensayo reciente demostró que la vacuna es muy eficaz para prevenir infecciones. Translated: A recent study showed that the vaccine is very effective at preventing infections.

Since these translations seem quite decent, we can now take the final step and come up with a few test examples in multiple languages, which we will pass to the pipeline for classification via prior translation:

# 4. PREDICTING ON UNSEEN DATA X_new_multilingual = [ "Un nuovo vaccino antinfluenzale ha mostrato un'efficacia dell'85% negli studi clinici, promettendo difese immunitarie più forti.", # Italian (Healthcare) "I ricercatori hanno sviluppato una batteria agli ioni di sodio che immagazzina l'energia eolica in eccesso in modo molto economico.", # Italian (Energy) "As ações de tecnologia se recuperaram no final do pregão, impulsionadas por fortes relatórios de lucros corporativos." # Portuguese (Finance) ] predictions = text_pipeline.predict(X_new_multilingual) new_translations = text_pipeline.named_steps['translator'].transform(X_new_multilingual) print("\n--- UNSEEN DATA PREDICTIONS ---") for text, translation, prediction in zip(X_new_multilingual, new_translations, predictions): print(f"Original Document: '{text}'") print(f"Intermediate Translation: '{translation}'") print(f"Final Predicted Category: {prediction}\n" + "-" * 70)

--- UNSEEN DATA PREDICTIONS --- Original Document: 'Un nuovo vaccino antinfluenzale ha mostrato un'efficacia dell'85% negli studi clinici, promettendo difese immunitarie più forti.' Intermediate Translation: 'A new influenza vaccine has shown an efficacy of 85% in clinical studies, promising stronger immune defenses.' Final Predicted Category: Healthcare ---------------------------------------------------------------------- Original Document: 'I ricercatori hanno sviluppato una batteria agli ioni di sodio che immagazzina l'energia eolica in eccesso in modo molto economico.' Intermediate Translation: 'Researchers have developed a sodium-ion battery that stores excess wind energy in a very economical way.' Final Predicted Category: Energy ---------------------------------------------------------------------- Original Document: 'As ações de tecnologia se recuperaram no final do pregão, impulsionadas por fortes relatórios de lucros corporativos.' Intermediate Translation: 'Technology stocks recovered at the end of the trading session, driven by strong corporate earnings reports.' Final Predicted Category: Finance ----------------------------------------------------------------------

The results speak for themselves, right? Well done!

Aided by Scikit-LLM classes for translation and zero-shot classification via pre-trained LLMs, we have learned how to build a free, quick, and highly effective two-stage scikit-learn pipeline for classifying texts in multiple languages.

Iván Palomares Carrascosa is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world.

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 KDnuggets(Integrating LLM Translation into Your Scikit-Learn Pipelines with Scikit-LLM),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗