Search: github.com/tokenzip | Heykuki News

HK

Heykuki News

Top New Best Ask Show Jobs

Top New Best Ask Show Jobs

31.

Show HN: Byte-Pair Encoding tokenizer for training LLMs on large datasets

github.com/jmaczan

2 years ago

5 points

32.

Show HN: 1gbps Tokenizer written in Assembly. 20x faster than HuggingFace

github.com/dogmaticdev

2 months ago

3 points

33.

Node.js Open Source LLM Tokenizer

github.com/jakecyr

2 years ago

2 points

34.

LLM Tokenization Demo

github.com/tokfan

10 months ago

2 points

35.

The Worst (But Only) Claude 3 Tokenizer

github.com/javirandor

2 years ago

2 points

36.

Neural Tokenizer

github.com/Kyubyong

9 years ago

2 points

37.

Claude-Tokenwise – CLI wrapper for efficient Claude token usage

github.com/nniinnoo

4 months ago

1 points

38.

Show HN: CLI Tokenizer – A tiny tool for prompt engineers

github.com/ericciarla

2 years ago

1 points

39.

Stripe on Apple watchOS 3

github.com/appintheair

10 years ago

4 points

40.

LLM Tokenizer in Zig

github.com/Mario-SO

10 months ago

1 points

41.

Very simple javascript highlighter that can be used in blog posts

github.com/fatih-erikli

2 years ago

1 points

42.

Show HN: I'm writing a library to apply NLP techniques to StarCraft 2

github.com/ZephyrBlu

5 years ago

1 points

43.

PRFI Protocol:Decentralized API Tokenization with Oof-of-Work Mining

github.com/sr-oliveiraa

a year ago

1 points

44.

Card Network Tokenization: A Savior or Hidden Menace

github.com/juspay

3 years ago

1 points

45.

Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

github.com/M4THYOU

a year ago

281 points

46.

Tiktoken: OpenAI’s Tokenizer

github.com/openai

4 years ago

153 points

47.

Code for the Byte Pair Encoding algorithm, commonly used in LLM tokenization

github.com/karpathy

2 years ago

81 points

48.

55x Speedup of Andrej Karpathy's Minbpe LLM Tokenizer with PyTorch/CUDA

github.com/kuprel

2 years ago

19 points

49.

Show HN: Open-source card tokenization service in Rust

github.com/juspay

3 years ago

14 points

50.

XML Tokenizer that's 4x faster than stdlib's XML

github.com/muktihari

2 years ago

10 points

51.

TokenMonster: Ungreedy tokenizer, outperforming tiktoken by 35%

github.com/alasdairforsythe

3 years ago

10 points

52.

Show HN: A Command-Line Sentence Tokenizer Written in Golang

github.com/neurosnap

11 years ago

6 points

53.

From Scratch GPT Built with NumPy (Tokenizer, Model, Adam)

github.com/codiceSpaghetti

a year ago

6 points

54.

Show HN: Rust BPE tokenizer for Qwen models that's 12x faster than HuggingFace

github.com/sweepai

9 months ago

5 points

55.

Chiffon: A very small ECMAScript parser, tokenizer in JS

github.com/polygonplanet

11 years ago

5 points

56.

Jargon: tokenizers and lemmatizers for Go

github.com/clipperhouse

8 years ago

4 points

57.

Fast JSON parser in Rust that uses SIMD and avoids tokenisation

github.com/pikkr

9 years ago

4 points

58.

VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and Voice Cloning

github.com/OpenBMB

7 months ago

3 points

59.

SSDD: Single-Step Diffusion Decoder for Efficient Image Tokenization

github.com/facebookresearch

9 months ago

3 points

60.

Show HN: Jsmn_Zig – Memory-Efficient JSON Tokenizer for Zig

9 months ago

3 points