Skip to content

Building without onig feature fails #1729

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
414owen opened this issue Jan 31, 2025 · 5 comments · May be fixed by #1772
Open

Building without onig feature fails #1729

414owen opened this issue Jan 31, 2025 · 5 comments · May be fixed by #1772

Comments

@414owen
Copy link

414owen commented Jan 31, 2025

cargo build --no-default-features fails with unresolved import onig.
If this isn't an optional feature, let's not mark it as such.
@414owen
Copy link
Author

414owen commented Jan 31, 2025

According to #1202

either it or fancy-regex is required

However

cargo build --no-default-features --features=fancy-regex

Also fails

@DCNick3
Copy link

DCNick3 commented Apr 9, 2025

I was able to build it by enabling unstable_wasm feature. Would be nice if just disabling onig and enabling fancy-regex would be enough

@xangelix
Copy link

xangelix commented May 1, 2025

This is especially important now that onig doesn't build on arch and fedora systems.

rust-onig/rust-onig#196

@414owen
Copy link
Author

414owen commented May 1, 2025

Agreed.

I'm on fedora, the workaround I've been using is to use export CC=gcc-14, when compiling anything that depends on tokenizers.

@414owen
Copy link
Author

414owen commented May 1, 2025

#1771

414owen added a commit to 414owen/tokenizers that referenced this issue May 1, 2025
@414owen
Fix no-onig no-wasm builds
bfdebfe 
Now, you can build with:

```
--no-default-features --features=fancy-regex
```

Which previously didn't work. You had to enable the `unstable_wasm`
flag.

I think using `fancy_regex` without wasm is a valid use-case, as I've
seen extremely slow build times using `onig`.
See: huggingface#1730

Onig also breaks, sometimes, with compiler updates.
See: huggingface#1771

Closes huggingface#1729
@414owen 414owen linked a pull request May 1, 2025 that will close this issue
Open
lucascherzer pushed a commit to lucascherzer/rocket-forum that referenced this issue May 22, 2025
fix: set env var CC to gcc-14
e323ef9 
This might fix the build on fedora systems:
huggingface/tokenizers#1729 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix no-onig no-wasm builds 414owen/tokenizers
3 participants
@414owen @DCNick3 @xangelix