Regular Expression Syntax
[All rights reserved: http://www.regexlab.com/en/regref.htm]
Introduction
Regular expression is to express a characteristic in a string, and then to match another string with the characteristic. For example, pattern "ab+" means "one 'a' and at least one 'b' ", so "ab", "abb", "abbbbbbb" match the pattern.
Regular expression is used to : (1) test a string whether it matches a pattern, such as a email address. (2) to find a substring which matches certain pattern, from a whole text. (3) to do complex replacement in a text.
It is very simple to study regular expression syntax, and the few abstract concepts can be understood easily too. Many articles does not introduce its concepts from simple ones to abstract ones step by step, so some persons may feel it is difficult
to study. On the other hand, each regular expression engine's document will describe its special function, but this part of special function is not what we should study first.
Every example in this article has a link to test page. Now let's begin!
1. Regular Expression Basic Syntax
1.1 Common Characters
Letters, numbers, the underline, and punctuations with no special definition are "common characters". When regular expression matches a string, a common character can match the same character.
 Example1: When pattern "c" matches string "abcde", match result: success; substring matched: "c"; position: starts at 2, ends at 3.
Example2: When pattern "bcd" matches string "abcde"，match result: success; substring matched: "bcd"; position: starts at 1, ends at 4.
1.2 Simple escaped characters
Nonprinting characters which we know:
Expression |
Matches |
\r, \n |
Carriage return, newline character |
\t |
Tabs |
\\ |
Matches "\" itself |
Some punctuations are specially defined in regular expression. To match these characters in string, add "\" in pattern. For example: ^, $ has special definition, so we need to use "\^" and "\$" to match them.
Expression |
Matches |
\^ |
Matches "^" itself |
\$ |
Matches "$" itself |
\. |
Matches dot(.) itself |
These escaped characters have the same effect as "common characters": to match a certain character.
Example1: When pattern "\$d" matches string "abc$de", match result: success; substring matched: "$d"; position: starts at 3, ends at 5.
1.3 Expression matches anyone of many characters
Some expressions can match anyone of many characters. For example: "\d" can match any number character. Each of these expressions can match only one character at one time, though they can match any character of a certain group of characters.
Expression |
Matches |
\d |
Any digit character, any one of 0~9 |
\w |
Any alpha, numeric, underline, any one of A~Z,a~z,0~9,_ |
\s |
Any one of space, tab, newline, return, or newpage character |
. |
'.' matches any character except the newline character(\n) |
Example1: When pattern "\d\d" matches "abc123", match result: success; substring matched: "12"; position: starts at 3, ends at 5.
Example2: When pattern "a.\d" matches "aaa100", match result: success; substring matched: "aa1"; position: starts at 1, ends at 4.
1.4 Custom expression matches anyone of many characters
Expression uses square brackets [ ] to contain a series of characters, it can match anyone of them. Uses [^ ] to contain a series of characters, it can match anyone character except characters contained.
Expression |
Matches |
[ab5@] |
Matches "a" or "b" or "5" or "@" |
[^abc] |
Matches any character except "a","b","c" |
[f-k] |
Any character among "f"~"k" |
[^A-F0-3] |
Any character except "A"~"F","0"~"3" |
Example1: When pattern "[bcd][bcd]" matches "abc123"
, match result: success; substring matched: "bc"; position: starts at 1, ends at 3.
Example2: When pattern "[^abc]" matches "abc123", match result: success; substring matched: "1"; position: starts at 3, ends at 4.
1.5 Special expression to quantify matching
All expressions introduced before can match character only one time. If a expression is followed by a quantifier, it can matches more than one times.
For example: we can use the pattern "[bcd]{2}" instead of "[bcd][bcd]".
Expression |
Function |
{n} |
Match exactly n times, example: "\w{2}" equals "\w\w"; "a{5}" equals "aaaaa" |
{m,n} |
At least m but no more than n times:
"ba{1,3}" matches "ba","baa","baaa" |
{m,} |
Match at least n times:
"\w\d{2,}" matches "a12","_456","M12344"... |
? |
Match 1 or 0 times, equivalent to {0,1}:
"a[cd]?" matches "a","ac","ad". |
+ |
Match 1 or more times, equivalent to {1,}:
"a+b" matches "ab","aab","aaab"... |
* |
Match 0 or more times, equivalent to {0,}:
"\^*b" matches "b","^^^b"... |
Example1: When pattern "\d+\.?\d*" matches "It costs $12.5", match result: success; substring matched:"12.5"; position: starts at 10, ends at 14.
Example2: When pattern "go{2,8}gle" matches "Ads by goooooogle", match result: success; substring matched: "goooooogle"; position: starts at 7, ends at 17.
1.6 Some special punctuations with abstract function
Some punctuations in pattern have special function:
Expression |
Function |
^ |
Match the beginning of the string |
$ |
Match the end of the string |
\b |
Match a word boundary |
More examples to help you to understand.
Example1: When pattern "^aaa" matches "xxx aaa xxx", match result: failed. Because "^" must match the beginning of the string. It could match successfully on condition that "aaa" is at the
beginning of the string, such as "aaa xxx xxx".
Example2: When pattern "aaa$" matches "xxx aaa xxx", match result: failed. Bacause "$" must match the end of the string. It could match successfully on condition that "aaa" is at the end of the
string, such as "xxx xxx aaa".
Example3: When pattern ".\b." matches "@@@abc", match result: success; substring matched: "@a"; position: starts at 2, ends at 4.
Further explanation: "\b" is similar to "^" and "$", matches no character itself, but it require a '\w' character at its one side, another not '\w' character at the other side.
Example4: When pattern "\bend\b" matches "weekend,endfor,end", match result: success; substring matched: "end"; position: starts at 15, ends at 18.
Some special punctuation can make effect on other sub-patterns:
Expression |
Function |
| |
Alternation, matches either left side or right side |
( ) |
(1). Let sub-patterns in it to be a whole part when it is quantified.
(2). Match result of sub-patterns in it can be retrieved individually |
Example5: When pattern "Tom|Jack" matches string "I'm Tom, he is Jack", match result: success; substring matched: "Tom"; position: starts at 4, ends at 7. When match next,
match result: success; substring matched: "Jack"; position: starts at 15, ends at 19.
Example6: When pattern "(go\s*)+" matches "Let's go go go!", match result: success; substring matched: "go go go"; position: starts at 6, ends at 14.
Example7: When pattern "￥(\d+\.?\d*)" matches "＄10.9,￥20.5", match result: success; substring matched: "￥20.5"; position: starts at 6, ends at 10. Match
result of sub-patterns in "( )" is: "20.5".
2. Regular expression advanced syntax
2.1 Reluctant or greedy quantifiers
There are serval method to quantify subpattern, such as: "{m,n}", "{m,}", "?", "*", "+". By default, a quantified subpattern is "greedy", that is, it will match as many times as possible (given a particular starting location) while still allowing
the rest of the pattern to match. For example, to match "dxxxdxxxd":
Expression |
Match result |
(d)(\w+) |
"\w+" matches all characters "xxxdxxxd" behind of "d" |
(d)(\w+)(d) |
"\w+" matches all characters "xxxdxxx" between the first "d" and the last "d". In order to let the whole pattern match success, "\w" has to give up the last "d", although it can match the last "d" too. |
Thus it can be seen that: when "\w+" matches, it will match as many characters as possible. In the second example, it does not match the last "d", but this is in order to let the whole pattern match successfully. Pattern with "*" or "{m,n}" will
also match as many times as possible, pattern with "?" will match if possible. This type of matching is called "greedy matching". 。
Reluctant Matching:
To follow the quantifier with a "?", it can let the pattern to match the minimum number of times possible. This type of matching is called reluctant matching. In order to let the whole pattern match successfully, the reluctant pattern may match a
few more times if it is required. For example, to match "dxxxdxxxd":
Expression |
Match result |
(d)(\w+?) |
"\w+?" match as few times as possible, so "\w+?" matches only one "x" |
(d)(\w+?)(d) |
In order to let the whole pattern match successfully, "\w+?" has to match "xxx". So, match result is: "\w+?" matches "xxx" |
More examples:
Example1: When pattern "<td>(.*)</td>" matches "<td><p>aa</p></td> <td><p>bb</p></td>", match result:
success; substring matched: the whole "<td><p>aa</p></td> <td><p>bb</p></td>", "</td>" in the pattern matches the last "</td>" in the string.
Example2: For comparison, when pattern "<td>(.*?)</td>" matches the string in example1, it matches
"<td><p>aa</p></td>". When match next, the next "<td><p>bb</p></td>" can be matched.
2.2 Referring to matched substring \1, \2...
During the process of matching, the match results of subpattern between parentheses "( )" are recorded for later use. When retrieving match results, those match result of subpattern can be retrieved individually, and this has been demonstrated
many times in former examples. In practice, parentheses "( )" must be used to get what we want indeed after match, such as "<td>(.*?)</td>".
In fact, those match result of subpattern between parentheses can be used not only after matching, but also during matching. The latter part of subpattern, can refer the match result of former subpattern. Usage: "\" plus a number to refer to the
corresponding substring. "\1" refers to 1st pair of parentheses' match result, "\2" refers to 2nd pair of parentheses' match result.
Examples:
Example1: When pattern "('|")(.*?)(\1)" matches " 'Hello', "World" ", match result: success; substring matched: " 'Hello' "; when match next, substring matched: " "World" ".
Example2: When pattern "(\w)\1{4,}" matches "aa bbbb abcdefg ccccc 111121111 999999999", match result: success; substring matched:
"ccccc"; when match next, substring matched "999999999". This pattern require a character of "\w" to repeat at least 5 times.
Pay attention to comparison with "\w{5,}".
Example3: When pattern "<(\w+)\s*(\w+(=('|").*?\4)?\s*)*>.*?</\1>" matches "<td id='td1' style="bgcolor:white"></td>", match result: success. If both "<td>" and "</td>" are not "td", the match will fail.
2.3 Lookahead assertion; Lookbehind assertion
In former chapters, I have introduced serval punctuations with special function: "^"，"$"，"\b". They all do not match any characters, but they all require certain conditions on their position. Now, this chapter will introduce more methods to add
conditions on the gap between characters.
Lookahead assertion: "(?=xxxxx)", "(?!xxxxx)"
Format: "(?=xxxxx)", the condition which it add on the gap is that: string on the right side of the gap must be abe to match the subpattern "xxxxx" between the parentheses. It is just a condition, not a match operation, so there is no match result.
Example1: When pattern "Windows (?=NT|XP)" matches "Windows 98, Windows NT, Windows 2000", it can match only "Windows " of "Windows NT", the
other "Windows " could not be matched.
Example2: When pattern "(\w)((?=\1\1\1)(\1))+" matches "aaa ffffff 999999999", it can match first 4 "f"s among the 6 "f"s, it can match
first 7 "9"s among 9 "9"s.
Format: "(?!xxxxx)", string on the right side of the gap must not be able to match the subpattern "xxxxx".
Example3: When pattern "((?!\bstop\b).)+" matches "fdjka ljfdl stop fjdsla fdj", it will match from the beginning of string to the position of "stop". If there is no "stop" in the string, the pattern will match the whole string.
Example4: When pattern "do(?!\w)" matches "done, do, dog", it can only match "do". Here, "(?!\w)" has the same effect as "\b".
Lookbehind assertion: "(?<=xxxxx)", "(?<!xxxxx)"
The concepts of "Lookbehind assertion" and "Lookahead assertion" are similar. "(?<=xxxxx)" and "(?<!xxxxx)" require the string on the left side of the gap to be able to match or to be not able to match the subpattern, not the right side. And they
will not match any characters themselves too.
Example5: When pattern "(?<=\d{4})\d+(?=\d{4})" matches "1234567890123456", it will match 8 numbers in the middle, except first 4 numbers and last 4 numbers. Because lookbehind assertion is not supported by JScript.RegExp, this example could not be
demonstrated. There are many engines support lookbehind assertion, such as java.util.regex package in Java 1.4 or later, System.Text.RegularExpressions namespace in .NET platform, and DEELX Regexp Engine etc.
3. Other usually supported rules
There are several usually supported rules which have not been mentioned.
3.1 In pattern, a character can be expressed as "\xXX" or "\uXXXX" ("X" is a hex number)
Format |
Character range |
\xXX |
0 ~ 255, such as space can be "\x20" |
\uXXXX |
Any character can be expressed as "\u" plus 4 hex numbers, such as "\u4E2D" |
3.2 While "\s", "\d", "\w", "\b" are specially defined, their uppercase letters have the opposite meaning
3.3 Specially defined characters table
Character |
Description |
^ |
Matches the beginning of the string. Use "\^" to match "^" itself |
$ |
Matches the end of the string. Use "\$" to match "$" itself |
( ) |
Grouping. Use "\(" and "\)" to match "(" and ")" |
[ ] |
Character class. Use "\[" and "\]" to match "[" and "]" |
{ } |
Define quantifiers. Use "\{" and "\}" to match "{" and "}" |
. |
Match any character except newline(\n). Use "\." to match "." itself |
? |
Let subpattern match 0 or 1 time. Use "\?" to match "?" itself |
+ |
Let subpattern match at least 1 times. Use "\+" to match "+" itself |
* |
Let subpattern match any times. Use "\*" to match "*" itself |
| |
Alternation. Use "\|" to match "|" itself |
3.4 If a subpattern is in "(?:xxxxx)", the match result is not recorded for later use.
Example1: When pattern "(?:(\w)\1)+" matches "a bbccdd efg", the substring matched: "bbccdd". The match result of subpattern in "(?:)" is not recorded, so "\1" is used
to refer to the match result of "(\w)".
3.5 Pattern attribute: Ignorecase，Singleline，Multiline，Global
Attribute |
Description |
Ignorecase |
Do case-insensitive pattern matching. Default is case-sensitive. |
Singleline |
Treat string as single line. That is, change "." to match any character whatsoever, even a newline, which it normally would not match. |
Multiline |
Treat string as multiple lines. Default is that "^" and "$" match at only the very start① of the string and end④ of the string. If multiline, they match the start③ of any line and end② of any line within the string:
①xxxxxxxxx②\n
③xxxxxxxxx④ |
Global |
Replace all matches if the pattern is used in replace operation. |
4. Integrated prompt
4.1 If you want to know what else are implemented by advanced engines, you can refer to DEELX Syntax on this site.
4.2 If the pattern is required to match the whole string, not a part of string, we may use "^" and "$", such as: "^\d+$" require the whole string consist of digit characters.
4.3 If the pattern is required to match a whole word, not a part of word, we may use "\b" at the beginning and the end of the pattern, such as:
use "\b(if|while|else|void|int……)\b" to match keywords in a program.
4.4 Do not let pattern match empty string "". Or you will get an empty substring matched, while the match operation returns success. For example: if we need a pattern to match "123"、"123."、"123.5"、".5", we should not use this pattern "\d*\.?\d*". Though there is
nothing, we may still get a success.
Proper pattern: "\d+\.?\d*|\.\d+".
4.5 Do not let a subpattern loop infinite times if the subpattern can match empty string.
4.6 Choose reluctant or greedy quantifier properly.
4.7 Only one side of "|" to match a certain character.
5. Relative Sponsored Links
[ DEELX Regexp Engine] - The most convenient regexp engine to use.
A new release of Regex Match Tracer, which will tell you where
in your regex failed to match:
[
Regex Match Tracer - v3.0] - new release of Regex Match
Tracer.
Author: sswater shi.
|
Regex Match Tracer - v2.1