MultiLabelBinarizer with pandas: One-Hot Encoding Stock Concept Tags for Similarity, Theme Averages and sklearn Pipelines

2026-11-17来源 dev.toEnglish

A stock usually carries several concept or theme tags at once: one name might be tagged "AI compute" and "chip packaging", another "energy storage", "grid upgrade" and "solid-state battery". Before you can compute anything across those tags (how many names sit in each theme, which names look alike, what the average activity of a theme is), you need them as a binary matrix: one row per stock, one column per tag, 1 where the tag applies.

scikit-learn's MultiLabelBinarizer does exactly that, and it pairs naturally with pandas. It also has a few behaviours that silently produce wrong matrices: strings treated as lists of characters, unknown tags dropped with only a warning, and column orders that change when the vocabulary changes. This tutorial covers all of them on a small table of placeholder tickers, then uses the matrix for tag counts, co-occurrence, stock-to-stock cosine similarity, rarity weights, theme-level averages and a reusable pipeline step.

The code runs on scikit-learn 1.9 and pandas 3.1.

The input: a list column

import numpy as np import pandas as pd from sklearn.preprocessing import MultiLabelBinarizer pd.set_option("display.width", 250) df = pd.DataFrame({ "ticker": ["S01", "S02", "S03", "S04", "S05", "S06", "S07"], "concepts": [ ["AI compute", "chip packaging"], ["humanoid robot", "AI compute"], ["energy storage"], ["energy storage", "grid upgrade", "solid-state battery"], ["AI compute"], [], # no tags yet ["humanoid robot", "servo motor"], ], }) 

Every cell is a Python list, including the empty one. That matters, and the section on pitfalls below shows why.

fit_transform into a labelled DataFrame

mlb = MultiLabelBinarizer() M = mlb.fit_transform(df["concepts"]) print(mlb.classes_) print(M.shape, M.dtype) onehot = pd.DataFrame(M, columns=mlb.classes_, index=df["ticker"]) print(onehot) 
['AI compute' 'chip packaging' 'energy storage' 'grid upgrade' 'humanoid robot' 'servo motor' 'solid-state battery'] (7, 7) int64 AI compute chip packaging energy storage grid upgrade humanoid robot servo motor solid-state battery ticker S01 1 1 0 0 0 0 0 S02 1 0 0 0 1 0 0 S03 0 0 1 0 0 0 0 S04 0 0 1 1 0 0 1 S05 1 0 0 0 0 0 0 S06 0 0 0 0 0 0 0 S07 0 0 0 0 1 1 0 

Three details to notice:

  1. classes_ is sorted alphabetically when you let the binarizer learn it. Column order is therefore determined by the vocabulary, not by the order tags appear in.
  2. The output is a plain NumPy array. Wrapping it in a DataFrame with columns=mlb.classes_ and the ticker index is what makes it usable with pandas.
  3. The stock with an empty list (S06) is kept as an all-zero row. Nothing disappears.

Column sums give the number of stocks per tag:

print(onehot.sum().sort_values(ascending=False)) 
AI compute 3 energy storage 2 humanoid robot 2 chip packaging 1 grid upgrade 1 servo motor 1 solid-state battery 1 dtype: int64 

Pitfall 1: strings become characters

MultiLabelBinarizer iterates over each sample. If a sample is a string rather than a list, iteration yields characters:

bad = MultiLabelBinarizer().fit(["AI compute", "energy storage"]) print(bad.classes_) 
[' ' 'A' 'I' 'a' 'c' 'e' 'g' 'm' 'n' 'o' 'p' 'r' 's' 't' 'u' 'y'] 

No error, just a 16-column matrix of letters. This happens most often when a single-tag cell was stored as a bare string, or when tags arrived as a delimited text field such as "AI compute; chip packaging". Normalise every cell before fitting:

def to_tag_list(x, sep=";"): if isinstance(x, str): return [t.strip() for t in x.split(sep) if t.strip()] if x is None or (isinstance(x, float) and np.isnan(x)): return [] return [str(t).strip() for t in x] df["concepts"] = df["concepts"].map(to_tag_list) 

Missing values need the same care: NaN is a float, and iterating over it raises a TypeError, so map it to an empty list.

Pitfall 2: unknown tags are dropped with a warning

A fitted binarizer only knows the classes it saw. Transforming a new sample that contains an unseen tag drops that tag:

import warnings with warnings.catch_warnings(record=True) as w: warnings.simplefilter("always") out = mlb.transform([["AI compute", "quantum"]]) print(out) print([str(x.message) for x in w]) 
[[1 0 0 0 0 0 0]] ["unknown class(es) ['quantum'] will be ignored"] 

In a batch job, that warning is easy to miss in the logs. Two options, depending on what you want:

  • Refit on the full vocabulary whenever the tag list changes, and accept that the column set changes too.
  • Fix the vocabulary up front with the classes parameter, so the column set is stable and new tags are a deliberate change.

Fixing the vocabulary with classes=

vocab = [ "AI compute", "chip packaging", "energy storage", "grid upgrade", "humanoid robot", "servo motor", "solid-state battery", "low-altitude economy", # in the vocabulary, not yet in the data ] mlb_fixed = MultiLabelBinarizer(classes=vocab) M2 = mlb_fixed.fit_transform(df["concepts"]) print(M2.sum(axis=0)) 
[3 1 2 1 2 1 1 0] 

With classes supplied, the columns follow your order exactly (not alphabetical), and a tag with no members yet still gets a column of zeros. This is the setting to use when the matrix feeds a model or a dashboard that expects the same columns every day. Comparing two snapshots of tag membership is also much simpler when both share one column layout.

Going back: inverse_transform

inverse_transform turns rows of the matrix back into tuples of tags, which is handy after you have filtered or edited the matrix:

print(mlb.inverse_transform(M[:3])) 
[('AI compute', 'chip packaging'), ('AI compute', 'humanoid robot'), ('energy storage',)] 

Tags come back in classes_ order, not in the original list order.

Sparse output for large vocabularies

Real tag vocabularies run into the hundreds or thousands, and each stock has only a few tags, so the dense matrix is mostly zeros. Ask for a sparse result:

mlb_sp = MultiLabelBinarizer(sparse_output=True) S = mlb_sp.fit_transform(df["concepts"]) print(type(S).__name__, S.nnz, S.shape) sdf = pd.DataFrame.sparse.from_spmatrix(S, index=df["ticker"], columns=mlb_sp.classes_) print(sdf.dtypes.iloc[0], round(sdf.sparse.density, 3)) 
csr_matrix 11 (7, 7) Sparse[int64, 0] 0.224 

Eleven non-zero entries out of 49, a density of 22%. On a vocabulary of a thousand tags with three or four tags per stock, density falls well below 1%, and the memory saving is large. Most scikit-learn estimators and sklearn.metrics.pairwise functions accept the CSR matrix directly, so you rarely need to densify.

Co-occurrence: which tags travel together

The matrix product of the transposed one-hot with itself counts, for every pair of tags, how many stocks carry both. The diagonal is the per-tag count.

co = onehot.T @ onehot print(co.loc[["AI compute", "humanoid robot", "energy storage"], ["AI compute", "humanoid robot", "energy storage", "grid upgrade"]]) 
 AI compute humanoid robot energy storage grid upgrade AI compute 3 1 0 0 humanoid robot 1 2 0 0 energy storage 0 0 2 1 

With sparse input, S.T @ S gives the same counts as a sparse matrix.

Stock-to-stock similarity

Rows of the one-hot matrix are tag vectors, so any vector similarity compares stocks by the themes they share. Cosine similarity is a common choice because it does not reward a stock merely for having many tags:

from sklearn.metrics.pairwise import cosine_similarity cs = pd.DataFrame(cosine_similarity(onehot.values), index=onehot.index, columns=onehot.index) print(cs.round(2)) 
ticker S01 S02 S03 S04 S05 S06 S07 ticker S01 1.00 0.50 0.00 0.00 0.71 0.0 0.0 S02 0.50 1.00 0.00 0.00 0.71 0.0 0.5 S03 0.00 0.00 1.00 0.58 0.00 0.0 0.0 S04 0.00 0.00 0.58 1.00 0.00 0.0 0.0 S05 0.71 0.71 0.00 0.00 1.00 0.0 0.0 S06 0.00 0.00 0.00 0.00 0.00 0.0 0.0 S07 0.00 0.50 0.00 0.00 0.00 0.0 1.0 

S03 and S04 share one tag; S04 has three, so their cosine is 1/√3 ≈ 0.58. S06 has no tags and is similar to nothing, including itself. scikit-learn returns 0 for an all-zero row rather than NaN, so filter untagged stocks out explicitly instead of reading their zeros as "dissimilar".

Down-weighting common tags

A tag that half the market carries says less about a stock than a narrow one. An inverse-document-frequency weight, the same idea used in text retrieval, expresses that:

n = len(onehot) idf = np.log((1 + n) / (1 + onehot.sum())) + 1 print(idf.round(3)) weighted = onehot * idf # broadcast per column cs_w = cosine_similarity(weighted.values) 
AI compute 1.693 chip packaging 2.386 energy storage 1.981 grid upgrade 2.386 humanoid robot 1.981 servo motor 2.386 solid-state battery 2.386 dtype: float64 

"AI compute", the most common tag here, gets the smallest weight. In the unweighted table, S01–S02 (sharing "AI compute") and S02–S07 (sharing "humanoid robot") tie at 0.50. With the weighted matrix they separate: S01–S02 drops to 0.38 while S02–S07 holds at 0.49, because the shared tag is narrower.

Theme-level averages from a per-stock metric

The one-hot matrix also acts as a membership mask for aggregations. Suppose you have a per-stock activity or attention score:

heat = pd.Series([82, 64, 40, 55, 71, 12, 48], index=onehot.index, name="heat") theme_mean = onehot.mul(heat, axis=0).sum() / onehot.sum() print(theme_mean.round(1).sort_values(ascending=False)) 
chip packaging 82.0 AI compute 72.3 humanoid robot 56.0 solid-state battery 55.0 grid upgrade 55.0 servo motor 48.0 energy storage 47.5 dtype: float64 

Each tag's value is the average score of the stocks carrying it. Keep the member count (onehot.sum()) next to it: "chip packaging" ranks first on a single stock, which is a sample size problem rather than a finding. Requiring a minimum member count before ranking themes avoids most of these artefacts.

How this compares to str.get_dummies

If tags are stored as one delimited string per row, pandas can do the encoding alone:

gd = df["concepts"].map("|".join).str.get_dummies("|") print((gd.values == onehot.values).all()) 
True 

Same result, also with sorted columns. get_dummies is convenient for quick exploration. MultiLabelBinarizer is better when you need a fixed vocabulary, sparse output, inverse_transform, or an object you can fit once and reuse on new data.

A reusable pipeline step

MultiLabelBinarizer is designed for target labels, so its fit takes a single argument and it does not plug straight into a ColumnTransformer. A small wrapper makes it a regular feature transformer, with a minimum-count filter for rare tags:

from sklearn.base import BaseEstimator, TransformerMixin class MultiLabelColumn(BaseEstimator, TransformerMixin): def __init__(self, column, min_count=1): self.column = column self.min_count = min_count def fit(self, X, y=None): counts = pd.Series([t for row in X[self.column] for t in set(row)]).value_counts() self.classes_ = sorted(counts[counts >= self.min_count].index) self.mlb_ = MultiLabelBinarizer(classes=self.classes_).fit([]) return self def transform(self, X): with warnings.catch_warnings(): warnings.simplefilter("ignore") # rare/unseen tags are dropped on purpose return self.mlb_.transform(X[self.column]) def get_feature_names_out(self, input_features=None): return np.array([f"{self.column}={c}" for c in self.classes_]) tr = MultiLabelColumn("concepts", min_count=2).fit(df) print(tr.get_feature_names_out()) print(tr.transform(df).sum(axis=0)) 
['concepts=AI compute' 'concepts=energy storage' 'concepts=humanoid robot'] [3 2 2] 

Fitted on training data, the step learns the vocabulary once; at prediction time the columns are identical, and tags that were rare or unseen in training are dropped intentionally rather than by surprise.

FAQ

Why are my columns single letters?
At least one sample is a string. Convert every cell to a list first.

Can I keep the tag order from my source system?
Pass classes= with the order you want. Without it, columns are sorted.

Does duplicate tagging produce a 2?
No. The output is binary, so ["AI compute", "AI compute"] still gives a single 1.

Conclusion

MultiLabelBinarizer with pandas turns a list column of concept tags into a labelled binary matrix in two lines. Make sure every cell is a real list, decide whether the vocabulary should be learned or fixed with classes=, watch for the unknown-class warning, and switch to sparse_output=True once the vocabulary grows. From there, tag counts, co-occurrence, cosine similarity between stocks, IDF weighting and theme-level averages are each one or two lines, and a thin wrapper makes the encoding a stable step in an sklearn pipeline.

If you need the underlying tag lists themselves, Lava's stock concepts lookup shows the concept tags attached to an individual A-share ticker.

中文简介:本文讲 scikit-learn 的 MultiLabelBinarizer 如何配合 pandas 把“股票-概念标签”列表列编码成 0/1 矩阵:说明 classes_ 默认按字母排序、空列表保留为全零行;指出字符串被拆成单个字符、NaN 报错、未见标签仅告警后丢弃等坑以及对应的规整写法;用 classes= 固定词表与列顺序,用 inverse_transform 还原、sparse_output 输出稀疏矩阵;再基于矩阵做概念共现、股票间余弦相似度、IDF 降权常见标签、按概念求成分股平均指标,并与 str.get_dummies 对比,最后封装成可复用的 sklearn 管道步骤。示例为占位数据,仅作方法演示。

原文:https://www.a8q9.cn/en/article/multilabelbinarizer-with-pandas-one-hot-encoding-stock-concept-tags-for-similari-1e11d1.html

相关文章