How I labeled a pile of unlabeled support tickets

Last Wednesday afternoon, a friend who does SaaS messaged me saying he had several thousand customer service tickets on hand and wanted to train a model to auto-triage them, and asked if I could find someone to label them.

He'd gotten a quote from an outsourcing shop—priced per item, and every time the category definitions changed, they'd re-label and charge again.

He went silent for half a minute. This really doesn't have to be done that way.

Whether a data-as-a-service product is worth using comes down to three conditions. Is there enough raw data—several hundred items or more; are the category definitions stable; and does the team have one person who can write the decision criteria in a single sentence. If all three are met, you can save a ton of repetitive labor. Conversely, if internally people can't even agree on "what counts as a complaint versus an inquiry," no tool will help. Tools solve efficiency; they can't solve fuzzy definitions.

I ran through Snorkel AI's data-as-a-service setup with a small scenario, splitting tickets into three categories: feature malfunction, billing issue, feature request. Its approach treats data as something you fetch on demand—you don't have to build a whole labeling system yourself; rules, sampling, and export all happen in one workspace. Here's the full path.

Step one: write the task as a single sentence.

This step is the easiest to skip and the easiest to screw up. I started out just writing "classify tickets," and everything went haywire afterward. The correct way to write it includes boundaries: any ticket that mentions a charge amount, invoice, or refund counts as a billing issue; any that describes a button doing nothing when clicked or a page error counts as a feature malfunction; only one that says it hopes to add some feature counts as a request.

Write it in a document, and whoever labels follows it. This step doesn't cost tool time—it costs your own time.

Step two: go into the workspace and create a project.

After registering and going in, there's a project list in the left column, and a new-project entry in the top right—pick the dataset project type. It'll ask you for a project name and a label set. For the project name, I'd suggest English with underscores, not Chinese—it saves hassle when exporting files later. The label set is the three categories above, one per line, no spaces.

Step three: upload the raw data.

It supports uploading CSV directly. From my testing, the most stable format is two columns, one id and one text. Use English headers and save as UTF-8. After uploading, the interface gives a preview—the first few dozen rows laid out so you can eyeball whether anything got misaligned.

The first pitfall is right here.

The first time I uploaded, the headers were Chinese—"ticket number" and "content description." The interface could read it in, but later when filtering and exporting, the fields all turned into garbled nonsense. Changing to English and re-uploading fixed it. Also, if there are unescaped newline characters in the CSV, one ticket gets split into two—you can spot it in the preview; if the row count doesn't match, that's the problem.

Step four: write rules first, don't rush to manual labeling.

This is where this thing differs most from traditional labeling. Traditionally you hire people to look one by one; here you first write a batch of decision rules and let the machine run through them first. A rule can be as simple as a keyword—if the text contains "refund," tag it as a billing issue; or it can be a regex matching an error-code format.

In the interface each rule is a card; after writing it, hit save, and it tells you in real time how many items that rule hit. Rules that hit too few are basically useless—delete them.

Rules will conflict with each other. A ticket that hits both "refund" and "hope to add an auto-renewal toggle"—is it billing or a request? There's a conflict-handling setting in the interface; I chose by priority, putting the more specific rules first. If you don't set this, conflicting items just come out blank—check the configuration first, don't rush to suspect the data.

Step five: sample a batch and label it yourself to see if the rules are accurate.

Rules finishing their run doesn't mean it's deliverable. The system has you sample for manual verification; I sampled one or two hundred and went through them one by one. Don't skip this step, and don't just sample the first few dozen. The first time I cut corners and sampled from the top—the front was all old tickets with regular formatting, and it looked pretty accurate; only when I got to the newer tickets did I find a bunch of colloquial expressions all missed. Sampling should be random—there's a random sampling option in the interface, check it.

After verification it gives you two numbers: coverage—how many items the rules hit—and accuracy—of those hit, how many were correct. From my run, coverage was just over half, and the rest that no rule hit automatically go into the manual queue. Low coverage doesn't mean the tool is bad—it usually means you wrote too few rules. After I added seven or eight rules targeting colloquial expressions, coverage clearly went up.

Step six: export.

Once you've confirmed it's fine, export. What comes out is a standard table with three columns: original text, final label, confidence. I'd suggest pulling out the low-confidence ones separately for another manual look—don't just take them straight to train a model.

I usually link this file with the original ticket table by id in Feishu Bitable to check for anything missing. I wrote a post before about data pipelines dropping fields—same principle: just leaving blanks doesn't help; you need a clean directory to enforce constraints, or dirty data still sneaks in.

Once you've learned this, the next step is to try running the same batch of data a second time—this time without writing rules, manually label fifty as seeds first, and see whether the system can suggest rules on its own. Compare the two approaches and you'll know which suits your data.

Running the whole workflow through, the tool part is actually small. Once that criteria document is written clearly, any tool will work.