module FuzzyMatch

Overview

Fuzzy string matching for Crystal.

Every scorer compares two strings and returns a similarity between 0.0 (nothing in common) and 100.0 (identical). The scorers follow the definitions used by Python's fuzzywuzzy and rapidfuzz, so scores carry over if you already have thresholds tuned for those libraries.

FuzzyMatch.ratio("this is a test", "this is a test!")                           # => 96.55
FuzzyMatch.partial_ratio("world", "hello world")                                # => 100.0
FuzzyMatch.token_sort_ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear") # => 100.0
FuzzyMatch.token_set_ratio("fuzzy was a bear", "fuzzy fuzzy was a bear")        # => 100.0

FuzzyMatch.extract_one("new york jets", teams).try &.choice # => "New York Jets"

See FuzzyMatch.ratio, FuzzyMatch.partial_ratio, FuzzyMatch.token_sort_ratio, FuzzyMatch.token_set_ratio, FuzzyMatch.weighted_ratio and FuzzyMatch.extract.

Defined in:

fuzzy_match.cr
fuzzy_match/extract.cr
fuzzy_match/normalize.cr
fuzzy_match/scorers.cr

Constant Summary

VERSION = "0.1.1"

Class Method Summary

Class Method Detail

def self.extract(query : String, choices : Enumerable(String), scorer : Scorer = :weighted_ratio, limit : Int32 | Nil = nil, cutoff : Float64 = 0.0) : Array(Match) #

Scores query against every string in choices and returns the matches sorted best first. Ties keep the order of choices.

  • scorer picks the scoring function; .weighted_ratio by default.
  • limit caps the number of matches returned.
  • cutoff drops every match scoring below it.
teams = ["Atlanta Falcons", "New York Jets", "New York Giants", "Dallas Cowboys"]

FuzzyMatch.extract("new york jets", teams, limit: 2)
# => [Match(@choice="New York Jets", @score=100.0, @index=1),
#     Match(@choice="New York Giants", @score=78.57, @index=2)]

FuzzyMatch.extract("cowboys", teams, scorer: :ratio, cutoff: 90.0) # => []

[View source]
def self.extract(query : String, choices : Enumerable(String), limit : Int32 | Nil = nil, cutoff : Float64 = 0.0, & : String, String -> Float64) : Array(Match) #

Same as .extract, but scores each pair with the block instead of a built-in scorer.

FuzzyMatch.extract("crystal", langs) { |a, b| FuzzyMatch.ratio(a, b, normalize: true) }

[View source]
def self.extract_one(query : String, choices : Enumerable(String), scorer : Scorer = :weighted_ratio, cutoff : Float64 = 0.0) : Match | Nil #

The best match for query in choices, or nil when nothing reaches cutoff. Takes the same arguments as .extract.

FuzzyMatch.extract_one("dallas", teams).try &.choice   # => "Dallas Cowboys"
FuzzyMatch.extract_one("vikings", teams, cutoff: 80.0) # => nil

[View source]
def self.extract_one(query : String, choices : Enumerable(String), cutoff : Float64 = 0.0, & : String, String -> Float64) : Match | Nil #

The best match for query in choices, or nil when nothing reaches cutoff. Takes the same arguments as .extract.

FuzzyMatch.extract_one("dallas", teams).try &.choice   # => "Dallas Cowboys"
FuzzyMatch.extract_one("vikings", teams, cutoff: 80.0) # => nil

[View source]
def self.normalize(s : String) : String #

Lowercases s, replaces every run of non-alphanumeric characters with a single space and trims the ends. This is what the token scorers and .weighted_ratio do to their input before comparing.

FuzzyMatch.normalize("  Hello, World! Øl & 123 ") # => "hello world øl 123"

Alphanumeric is decided per Unicode character, so letters outside ASCII (æ, ø, å, ü, ...) are kept and only punctuation and symbols become spaces.


[View source]
def self.partial_ratio(a : String, b : String, normalize : Bool = false) : Float64 #

Best .ratio of the shorter string against any window of the longer one.

Useful when one string is a fragment of the other: a surname against a full name, a word against a sentence. The windows include ones that hang over the start or end of the longer string, so a partially matching prefix or suffix still counts.

FuzzyMatch.partial_ratio("world", "hello world")               # => 100.0
FuzzyMatch.partial_ratio("Lillevang", "Lillevang Consult ApS") # => 100.0
FuzzyMatch.partial_ratio("kitten", "sitting")                  # => 66.67

[View source]
def self.partial_token_set_ratio(a : String, b : String, normalize : Bool = true) : Float64 #

Like .token_set_ratio, but compares the leftovers with .partial_ratio. Any word in common scores 100.0.


[View source]
def self.partial_token_sort_ratio(a : String, b : String, normalize : Bool = true) : Float64 #

Like .token_sort_ratio, but compares with .partial_ratio so a shorter string whose words all appear in the longer one scores 100.0.


[View source]
def self.ratio(a : String, b : String, normalize : Bool = false) : Float64 #

Plain similarity of the two strings as a whole.

Defined as 2 * LCS / (|a| + |b|) * 100, where LCS is the length of the longest common subsequence. Equivalently, it is the normalized indel distance: how many insertions and deletions turn one string into the other, relative to the combined length. Two empty strings score 100.0.

FuzzyMatch.ratio("this is a test", "this is a test!")   # => 96.55
FuzzyMatch.ratio("kitten", "sitting")                   # => 61.54
FuzzyMatch.ratio("Crystal", "crystal")                  # => 85.71
FuzzyMatch.ratio("Crystal", "crystal", normalize: true) # => 100.0

[View source]
def self.token_set_ratio(a : String, b : String, normalize : Bool = true) : Float64 #

Similarity based on the sets of words in each string.

The words both strings share are split off into an intersection, and the leftovers of each string are compared with and without that common prefix. Duplicated words and word order are ignored, and a string whose words are all contained in the other scores 100.0. This is the scorer for "is this the same thing, written differently?".

FuzzyMatch.token_set_ratio("fuzzy was a bear", "fuzzy fuzzy was a bear") # => 100.0
FuzzyMatch.token_set_ratio("mariners vs angels",
  "los angeles angels of anaheim at seattle mariners") # => 90.91

Returns 0.0 when either string has no words at all.


[View source]
def self.token_sort_ratio(a : String, b : String, normalize : Bool = true) : Float64 #

.ratio of the two strings with their words sorted alphabetically, so word order no longer matters.

FuzzyMatch.token_sort_ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear") # => 100.0
FuzzyMatch.token_sort_ratio("New York Mets vs Atlanta Braves",
  "Atlanta Braves vs New York Mets") # => 100.0

[View source]
def self.weighted_ratio(a : String, b : String, normalize : Bool = true) : Float64 #

The all-rounder: picks the scorer that fits the two strings.

Strings of similar length are compared with .ratio and the token scorers. When one string is much longer than the other, the partial scorers are used instead, and their scores are scaled down so that an exact match still ranks above a fragment match. The input is normalized first. This is the default scorer for .extract.

FuzzyMatch.weighted_ratio("New York Jets", "new york jets") # => 100.0
FuzzyMatch.weighted_ratio("Jets", "New York Jets")          # => 90.0
FuzzyMatch.weighted_ratio("kitten", "sitting")              # => 61.54

Returns 0.0 when either string is empty after normalization.


[View source]