Skip to content

Update NER #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 24, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 29 additions & 4 deletions src/content/docs/reference/preparation/ner.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,36 @@
---
title: Named Entity Recognition
description: A guide in my new Starlight docs site.
description: A concise guide to Named Entity Recognition methods, applications, and challenges.
---

Guides lead a user through a specific task they want to accomplish, often with a sequence of steps.
Writing a good guide requires thinking about what your users are trying to do.
Named Entity Recognition (NER) is the process of automatically identifying and categorizing key elements in text such as names of people, organizations, locations, and more.
It is a foundational technique in natural language processing (NLP) that facilitates a range of applications including information extraction, content classification, and search optimization.

## Traditional NER Methods

Early NER systems relied on rule-based and statistical approaches.
Tools like [SpaCy](https://spacy.io/) implement a combination of hand-crafted rules and machine learning algorithms to efficiently recognize entities in text.
However, traditional NER models are typically effective only for a set of predefined entity types, limiting their flexibility.

### GliNER: A Compact, Flexible Alternative

In contrast to conventional systems, GliNER introduces a compact NER model designed to identify any type of entity.
Leveraging a bidirectional transformer encoder, GliNER facilitates parallel entity extraction—an advantage over the slow sequential token generation characteristic of many large language models (LLMs).
Comprehensive testing shows that GliNER outperforms both ChatGPT and fine-tuned LLMs in zero-shot evaluations on various NER benchmarks, addressing limitations in traditional models while maintaining resource efficiency.

## NER with Large Language Models

Modern large language models (LLMs) have introduced a paradigm shift in how NER tasks can be approached.
LLMs can perform NER in a zero-shot or few-shot learning setting, extracting arbitrary entities through natural language instructions.
This offers greater flexibility compared to traditional methods, although the size and cost of LLMs can be impractical in resource-limited scenarios.

## Entity Linking and Entity Disambiguation

Beyond merely identifying entities, many applications require associating them with specific, unique identifiers—a process known as entity linking.
This often involves resolving ambiguities where the same name might refer to multiple real-world entities, a challenge known as entity disambiguation.
High-level solutions typically integrate NER with external knowledge bases and context-aware algorithms to ensure that each recognized entity is accurately matched to its correct reference.

## Further reading

- Read [about how-to guides](https://diataxis.fr/how-to-guides/) in the Diátaxis framework
- [GliNER paper](https://arxiv.org/abs/2311.08526)
- [SpaCy](https://spacy.io/)