Lilac

Semantic and keyword search across your dataset is the core of Lilac, an open-source data tool for people preparing text for large language models.

Lilac lets you search, quantify and edit data so you can see what is in a training set before it reaches a model.

Beyond search, it can detect PII, flag duplicates, identify the language of each entry, and compare or edit fields side by side.

You can also plug in your own custom signals, and use fuzzy-concept search with refinement to find examples of an idea that keywords alone would miss.

The focus is data quality, so Lilac suits data and AI practitioners who want to clean and inspect datasets rather than build models.

It is open source, which means you can run and inspect it yourself.