How Did Open Food Facts Fix OCR-Extracted Ingredients Using Open-Source LLMs? | by Jeremy Arancio | Oct, 2024

Open Meals Info has tried to resolve this challenge for years utilizing Common Expressions and present options reminiscent of Elasticsearch’s corrector, with out success. Till not too long ago.

Due to the most recent developments in synthetic intelligence, we now have entry to highly effective Massive Language Fashions, additionally known as LLMs.

By coaching our personal mannequin, we created the Components Spellcheck and managed to not solely outperform proprietary LLMs reminiscent of GPT-4o or Claude 3.5 Sonnet on this job, but in addition to cut back the variety of unrecognized components within the database by 11%.

This text walks you thru the completely different levels of the undertaking and reveals you ways we managed to enhance the standard of the database utilizing Machine Studying.

Benefit from the studying!

When a product is added by a contributor, its footage undergo a sequence of processes to extract all related data. One essential step is the extraction of the record of components.

When a phrase is recognized as an ingredient, it’s cross-referenced with a taxonomy that accommodates a predefined record of acknowledged components. If the phrase matches an entry within the taxonomy, it’s tagged as an ingredient and added to the product’s data.

This tagging course of ensures that components are standardized and simply searchable, offering correct knowledge for shoppers and evaluation instruments.

But when an ingredient isn’t acknowledged, the method fails.

For that reason, we launched an extra layer to the method: the Components Spellcheck, designed to appropriate ingredient lists earlier than they’re processed by the ingredient parser.

An easier method could be the Peter Norvig algorithm, which processes every phrase by making use of a sequence of character deletions, additions, and replacements to determine potential corrections.

Nonetheless, this methodology proved to be inadequate for our use case, for a number of causes:

• Particular Characters and Formatting: Components like commas, brackets, and proportion indicators maintain essential significance in ingredient lists, influencing product composition and allergen labeling (e.g., “salt (1.2%)”).
• Multilingual Challenges: the database accommodates merchandise from all around the phrase with all kinds of languages. This additional complicates a primary character-based method like Norvig’s, which is language-agnostic.

As an alternative, we turned to the most recent developments in Machine Studying, significantly Massive Language Fashions (LLMs), which excel in all kinds of Pure Language Processing (NLP) duties, together with spelling correction.

That is the trail we determined to take.

You possibly can’t enhance what you don’t measure.

What is an efficient correction? And the right way to measure the efficiency of the corrector, LLM or non-LLM?

Our first step is to grasp and catalog the variety of errors the Ingredient Parser encounters.

Moreover, it’s important to evaluate whether or not an error ought to even be corrected within the first place. Generally, making an attempt to appropriate errors may do extra hurt than good:

flour, salt (1!2%)
# Is it 1.2% or 12%?...

For these causes, we created the Spellcheck Pointers, a algorithm that limits the corrections. These tips will serve us in some ways all through the undertaking, from the dataset technology to the mannequin analysis.

The rules was notably used to create the Spellcheck Benchmark, a curated dataset containing roughly 300 lists of components manually corrected.

This benchmark is the cornerstone of the undertaking. It permits us to judge any answer, Machine Studying or easy heuristic, on our use case.

It goes alongside the Analysis algorithm, a customized answer we developed that rework a set of corrections into measurable metrics.

The Analysis Algorithm

A lot of the present metrics and analysis algorithms for text-relative duties compute the similarity between a reference and a prediction, reminiscent of BLEU or ROUGE scores for language translation or summarization.

Nonetheless, in our case, these metrics fail quick.

We need to consider how nicely the Spellcheck algorithm acknowledges and fixes the precise phrases in a listing of components. Subsequently, we adapt the Precision and Recall metrics for our job:

Precision = Proper corrections by the mannequin / ​Whole corrections made by the mannequin

Recall = Proper corrections by the mannequin / ​Whole variety of errors

Nonetheless, we don’t have the fine-grained view of which phrases had been imagined to be corrected… We solely have entry to:

• The authentic: the record of components as current within the database;
• The reference: how we anticipate this record to be corrected;
• The prediction: the correction from the mannequin.

Is there any strategy to calculate the variety of errors that had been appropriately corrected, those that had been missed by the Spellcheck, and eventually the errors that had been wrongly corrected?

The reply is sure!

Unique:       "Th cat si on the fride,"
Reference:      "The cat is on the fridge."
Prediction:     "Th large cat is within the fridge."

With the instance above, we are able to simply spot which phrases had been imagined to be corrected: The , is and fridge ; and which phrases had been wrongly corrected: on into in. Lastly, we see that an extra phrase was added: large .

If we align these 3 sequences in pairs, original-reference and original-prediction , we are able to detect which phrases had been imagined to be corrected, and those who weren’t. This alignment drawback is typical in bio-informatic, known as Sequence Alignment, whose objective is to determine areas of similarity.

It is a good analogy for our spellcheck analysis job.

Unique:       "Th    -   cat   si   on   the   fride,"
Reference:      "The   -   cat   is   on   the   fridge."
1    0    0    1    0    0     1
Unique:       "Th    -   cat   si   on   the   fride,"
Prediction:     "Th   large  cat   is   in   the   fridge."
0    1    0    1    1    0     1
FN    FP         TP   FP         TP

By labeling every pair with a 0 or 1 whether or not the phrase modified or not, we are able to calculate how typically the mannequin appropriately fixes errors (True Positives — TP), incorrectly adjustments appropriate phrases (False Positives — FP), and misses errors that ought to have been corrected (False Negatives — FN).

In different phrases, we are able to calculate the Precision and Recall of the Spellcheck!

We now have a sturdy algorithm that’s able to evaluating any Spellcheck answer!

You will discover the algorithm within the project repository.

Massive Language Fashions (LLMs) have proved being nice assist in tackling Pure Language job in varied industries.

They represent a path we’ve got to probe for our use case.

Many LLM suppliers brag concerning the efficiency of their mannequin on leaderboards, however how do they carry out on correcting error in lists of components? Thus, we evaluated them!

We evaluated GPT-3.5 and GPT-4o from OpenAI, Claude-Sonnet-3.5 from Anthropic, and Gemini-1.5-Flash from Google utilizing our customized benchmark and analysis algorithm.

We prompted detailed directions to orient the corrections in direction of our customized tips.

GPT-3.5-Turbo delivered the perfect efficiency in comparison with different fashions, each by way of metrics and handbook overview. Particular point out goes to Claude-Sonnet-3.5, which confirmed spectacular error corrections (excessive Recall), however typically offered further irrelevant explanations, decreasing its Precision.

Nice! We’ve an LLM that works! Time to create the function within the app!

Properly, not so quick…

Utilizing personal LLMs reveals many challenges:

1. Lack of Possession: We turn into depending on the suppliers and their fashions. New mannequin variations are launched continuously, altering the mannequin’s habits. This instability, primarily as a result of the mannequin is designed for normal functions moderately than our particular job, complicates long-term upkeep.
2. Mannequin Deletion Threat: We’ve no safeguards towards suppliers eradicating older fashions. As an illustration, GPT-3.5 is slowly being exchange by extra performant fashions, regardless of being the perfect mannequin for this job!
3. Efficiency Limitations: The efficiency of a non-public LLM is constrained by its prompts. In different phrases, our solely approach of bettering outputs is thru higher prompts since we can’t modify the core weights of the mannequin by coaching it on our personal knowledge.

For these causes, we selected to focus our efforts on open-source options that would supply us with full management and outperform normal LLMs.

Any machine studying answer begins with knowledge. In our case, knowledge is the corrected lists of components.

Nonetheless, not all lists of components are equal. Some are freed from unrecognized components, some are simply so unreadable they might be no level correcting them.

Subsequently, we discover a good stability by selecting lists of components having between 10 and 40 p.c of unrecognized components. We additionally ensured there’s no duplicate inside the dataset, but in addition with the benchmark to stop any knowledge leakage in the course of the analysis stage.

We extracted 6000 uncorrected lists from the Open Meals Info database utilizing DuckDB, a quick in-process SQL device able to processing tens of millions of rows beneath the second.

Nonetheless, these extracted lists usually are not corrected but, and manually annotating them would take an excessive amount of time and assets…

Nonetheless, we’ve got entry to LLMs we already evaluated on the precise job. Subsequently, we prompted GPT-3.5-Turbo, the perfect mannequin on our benchmark, to appropriate each record in respect of our tips.

The method took lower than an hour and price practically 2$.

We then manually reviewed the dataset utilizing Argilla, an open-source annotation device specialised in Pure Language Processing duties. This course of ensures the dataset is of adequate high quality to coach a dependable mannequin.

We now have at our disposal a training dataset and an evaluation benchmark to coach our personal mannequin on the Spellcheck job.

Coaching

For this stage, we determined to go along with Sequence-to-Sequence Language Fashions. In different phrases, these fashions take a textual content as enter and returns a textual content as output, which fits the spellcheck course of.

A number of fashions match this function, such because the T5 household developed by Google in 2020, or the present open-source LLMs reminiscent of Llama or Mistral, that are designed for textual content technology and following directions.

The mannequin coaching consists in a succession of steps, each requiring completely different assets allocations, reminiscent of cloud GPUs, knowledge validation and logging. For that reason, we determined to orchestrate the coaching utilizing Metaflow, a pipeline orchestrator designed for Knowledge science and Machine Studying initiatives.

The coaching pipeline consists as observe:

• Configurations and hyperparameters are imported to the pipeline from config yaml information;
• The coaching job is launched within the cloud utilizing AWS Sagemaker, alongside the set of mannequin hyperparameters and the customized modules such because the analysis algorithm. As soon as the job is finished, the mannequin artifact is saved in an AWS S3 bucket. All coaching particulars are tracked utilizing Comet ML;
• The fine-tuned mannequin is then evaluated on the benchmark utilizing the analysis algorithm. Relying on the mannequin sizem this course of may be extraordinarily lengthy. Subsequently, we used vLLM, a Python library designed to accelerates LLM inferences;
• The predictions towards the benchmark, additionally saved in AWS S3, are despatched to Argilla for human-evaluation.

After iterating again and again between refining the information and the mannequin coaching, we achieved efficiency corresponding to proprietary LLMs on the Spellcheck job, scoring an F1-Rating of 0.65.