python – lower() vs. casefold() in string matching and converting to lowercase
- Purely ASCII Text ->
- Unicode text/user input ->
Casefolding is a more aggressive version of
lower() that is set up to make many of the more unique unicode characters more comparable. It is another form of normalizing text that may initially appear to be very different, but it takes characters of many different languages into account.
I suggest you take a closer look into what case folding actually is, so heres a good start: W3 Case Folding Wiki
To answer your other two questions, if you are working strictly in the English language,
casefold() should be yielding exactly the same results.
However, if you are trying to normalize text from other languages that use more than our simple 26-letter alphabet (using only ASCII), I would use
casefold() to compare your strings, as it will yield more consistent results.
Elastic.co Case Folding
Edit: I just recently found another very good related answer to a slightly different question here on SO (doing a case-insensitive string comparison)
Another Edit: @Voos comments have been bouncing around in the back of my mind for a few months, so here are some further thoughts:
As Voo mentioned, there arent any languages that never use text outside the standard ASCII values. Thats pretty much why Unicode exists. With that in mind, it makes more sense to me to use
casefold() on anything that is user-entered that can contain non-ascii values. This might end up excluding some text that might come from a database that strictly deals with ASCII, but, in general, probably most user input would be dealt with using
casefold() because it has the logic to properly de-uppercase all of the characters.
On the other hand, values that are known to be generated into the ASCII character space like hex UUIDs or something like that should be normalized with
lower() because it is a much simpler transformation. Simply put,
lower() will require less memory or less time because there are no lookups, and its only dealing with 26 characters it has to transform. Additionally, if you know that the source of your information is coming from a
VARCHAR (SQL Server fields) database field, you can similarly just use
lower because Unicode characters cant be entered into those fields.
So really, this question comes down to knowing the source of your data, and when in doubt about your user-entered information, just
lower() vs casefold() and when to use, Details are given below.