String processing is fairly easy in Stata because of the many built-in string functions. Among these string functions are three functions that are related to regular expressions, regexm for matching, regexr for replacing and regexs for subexpressions. We will show some examples of how to use regular expression to extract and/or replace a portion of a string variable using these three functions. At the bottom of the page is an explanation of all the regular expression operators as well as the functions that work with regular expressions.
Examples
Example 1: A researcher has addresses as a string variable and wants to create a new variable that contains just the zip codes.
Example 2: We have a variable that contains full names in the order of first name and then last name. We want to create a new variable with full name in the order of last name and then first name separated by comma.
Example 2: Dates were entered as a string variable, in some cases the year was entered as a four-digit value (which is what Stata generally expects to see), but in other cases it was entered as a two-digit value. We want to create a date variable in numeric format based on this string variable. This task can actually easily be handled with regular Stata commands, see our FAQ page “My date variable is a string, how can I turn it into a date variable Stata can recognize?” for information on doing this. We have included this example here for demonstration purposes, not because regular expressions are necessarily the best way to handle this situation.
In these situations, regular expressions can be used to identify cases in which a string contains a set of values (e.g. a specific word, a number followed by a word etc.) and extract that set of values from the whole string for use elsewhere.
Example 1: Extracting zip codes from addresses
Let’s start with some fake entries of addresses.
input str60 address "4905 Lakeway Drive, College Station, Texas 77845 USA" "673 Jasmine Street, Los Angeles, CA 90024" "2376 First street, San Diego, CA 90126" "6 West Central St, Tempe AZ 80068" "1234 Main St. Cambridge, MA 01238-1234" end
To find the zip code we will look for a five-digit number within an address. The gen command (short for "generate") below tells Stata to generate a new variable called zip. The rest of the command is a little tricky, the "if" is evaluated first, if(regexm(address, “[0-9][0-9][0-9][0-9][0-9]”)) searches the variable address for a five digit number, and, if it can find a five digit number in the variable address, the = regexs(0) indicates that Stata should set the value of zip to be equal to that five-digit number. We indicate that we want a five-digit number by specifying “[0-9]” five times. Unless otherwise indicated using a *, +, or ? mark, one and only one of the characters contained in brackets will be matched. This means that stringing five of these expressions together will enable us to find a string of exactly five digits. Note that the 0-9 indicates that the expression should match any character 0 through 9 (i.e. 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9 are all matches).
gen zip = regexs(0) if(regexm(address, "[0-9][0-9][0-9][0-9][0-9]"))
list
+--------------------------------------------------------------+
| address zip |
|--------------------------------------------------------------|
1. | 4905 Lakeway Drive, College Station, Texas 77845 USA 77845 |
2. | 673 Jasmine Street, Los Angeles, CA 90024 90024 |
3. | 2376 First street, San Diego, CA 90126 90126 |
4. | 6 West Central St, Tempe AZ 80068 80068 |
5. | 1234 Main St. Cambridge, MA 01238-1234 01238 |
+--------------------------------------------------------------+
Example 1, Variation Number 1
In our simplified example above, none of the addresses have five-digit street numbers. What if there are addresses with five-digit street numbers? Let’s look at another dataset of fake addresses and see what happens when we try to use the same code above.
clear
input str60 address
"4905 Lakeway Drive, College Station, Texas 77845"
"673 Jasmine Street, Los Angeles, CA 90024"
"2376 First street, San Diego, CA 90126"
"66666 West Central St, Tempe AZ 80068"
"12345 Main St. Cambridge, MA 01238"
end
gen zip = regexs(0) if(regexm(address, "[0-9][0-9][0-9][0-9][0-9]"))
list
+----------------------------------------------------------+
| address zip |
|----------------------------------------------------------|
1. | 4905 Lakeway Drive, College Station, Texas 77845 77845 |
2. | 673 Jasmine Street, Los Angeles, CA 90024 90024 |
3. | 2376 First street, San Diego, CA 90126 90126 |
4. | 66666 West Central St, Tempe AZ 80068 66666 |
5. | 12345 Main St. Cambridge, MA 01238 12345 |
+----------------------------------------------------------+
Apparently, this is not working correctly since the last two rows of the variable zip have picked up the street numbers for these addresses instead of zip codes. In this data set, the zip code appears at the end of the address string. If we assume that this the case for all addresses in the data, the remedy will be really simple. We can specify "[0-9][0-9][0-9][0-9][0-9]$" which would instruct Stata to find a five-digit number at the end of the string.
gen zip = regexs(0) if(regexm(address, "[0-9][0-9][0-9][0-9][0-9]$"))
list
+----------------------------------------------------------+
| address zip |
|----------------------------------------------------------|
1. | 4905 Lakeway Drive, College Station, Texas 77845 77845 |
2. | 673 Jasmine Street, Los Angeles, CA 90024 90024 |
3. | 2376 First street, San Diego, CA 90126 90126 |
4. | 66666 West Central St, Tempe AZ 80068 80068 |
5. | 12345 Main St. Cambridge, MA 01238 01238 |
+----------------------------------------------------------+
Example 1, Variation Number 2
Sometimes zip code also include the four-digit extension and the country name may also appear at the end of the address, such as in some of the addresses shown below.
clear input str60 address "4905 Lakeway Drive, College Station, Texas 77845 USA" "673 Jasmine Street, Los Angeles, CA 90024" "2376 First street, San Diego, CA 90126" "66666 West Central St, Tempe AZ 80068" "12345 Main St. Cambridge, MA 01238-1234" "12345 Main St Sommerville MA 01239-2345" "12345 Main St Watertwon MA 01239 USA" end
In this type of more realistic situation, the code in the previous examples won’t work correctly since there are extra characters after the zip code to be extracted. Here is how we can do it using a more complicated regular expression.
gen zip = regexs(1) if regexm(address, "([0-9][0-9][0-9][0-9][0-9])[-]*[0-9]*[ a-zA-Z]*$")list+--------------------------------------------------------------+ | address zip | |--------------------------------------------------------------| 1. | 4905 Lakeway Drive, College Station, Texas 77845 USA 77845 | 2. | 673 Jasmine Street, Los Angeles, CA 90024 90024 | 3. | 2376 First street, San Diego, CA 90126 90126 | 4. | 66666 West Central St, Tempe AZ 80068 80068 | 5. | 12345 Main St. Cambridge, MA 01238-1234 01238 | |--------------------------------------------------------------| 6. | 12345 Main St Sommerville MA 01239-2345 01239 | 7. | 12345 Main St Watertwon MA 01239 USA 01239 | +--------------------------------------------------------------+
What we have added in the regular expression is this sub-: "[-]*[0-9]*[ a-zA-Z]*". There are three components in this regular expression.
- [-]* – matching zero or more dashes "-"
- [0-9]* – matching zero or more numbers
- [ a-zA-Z]* – matching zero or more blank spaces or letters
These additions allow us to match up the cases where there are trailing characters after the zip code and to extract the zip code correctly. Notice that we also used "regexs(1)" instead of "regexs(0)" as we did previously, because we are now using subexpressions indicated by the pair of parenthesis in "([0-9][0-9][0-9][0-9][0-9])". Another strategy that might work better in some cases is the regular expression
gen zip2 = regexs(1) if(regexm(address, ".*([0-9][0-9][0-9][0-9][0-9])"))
In this example, the period (i.e. “.”) matches any charctor, and the asterix alone (“*”) matches any characters. Together, the two indicate that the number we are looking for should not occur at the very beginning of the string, but may occur anywhere after.
Example 2: Extracting first name and last name and switching their order
We have a variable that contains a person’s full name in the order of first name and then last name. We want to create a new variable for full name in the order of last name and then first name separated by comma. To start, let’s make a sample data set.
clear input str40 fullname "John Adams" "Adam Smiths" "Mary Smiths" "Charlie Wade" end
Now we need to capture the first word and the second word and swap them. Here is the regular expression for this purpose: (([a-zA-Z]+)[ ]*([a-zA-Z]+)).
There are three parts in this regular expression:
- ([a-zA-Z]+) – subexpression capturing a string consisting of letters, both lower case and upper case. This will be the first name.
- [ ]* – matching with space(s). This is the spacing between first name and last name.
- ([a-zA-Z]+) – subexpression capturing a string consisting of letters. This will be the last name.
gen n = regexs(2)+", "+regexs(1) if regexm(fullname, "([a-zA-Z]+)[ ]*([a-zA-Z]+)")
list
+------------------------------+
| fullname n |
|------------------------------|
1. | John Adams Adams, John |
2. | Adam Smiths Smiths, Adam |
3. | Mary Smiths Smiths, Mary |
4. | Charlie Wade Wade, Charlie |
+------------------------------+
This indeed works. Let’s see how regexs works in this case. regex actually identifies a number of sections, based on the whole expression as well as the subexpressions. The following code uses regexs to place each of these components (subexpressions) into its own variable and then displays them.
gen n0 = regexs(0) if regexm(fullname, "(([a-zA-Z]+)[ ]*([a-zA-Z]+))")
gen n1 = regexs(2) if regexm(fullname, "(([a-zA-Z]+)[ ]*([a-zA-Z]+))")
gen n2 = regexs(3) if regexm(fullname, "(([a-zA-Z]+)[ ]*([a-zA-Z]+))")
list fullname n0 n1 n2
+------------------------------------------------+
| fullname n0 n1 n2 |
|------------------------------------------------|
1. | John Adams John Adams John Adams |
2. | Adam Smiths Adam Smiths Adam Smiths |
3. | Mary Smiths Mary Smiths Mary Smiths |
4. | Charlie Wade Charlie Wade Charlie Wade |
+------------------------------------------------+
Example 3: Two- and four- digit values for year.
In this example, we have dates entered as a string variable. Stata can handle this using standard commands (see "My date variable is a string, how can I turn it into a date variable Stata can recognize?"), we are using this as an example of what you could do with regular expressions. The goal of this process is to produce a string variable with the appropriate four digit year for every case, which Stata can then easily convert into a date. To do this we will start by separating out each element of the date (day, month, and two- or four- digit year) into a separate variable, then we will assign the correct four-digit year to cases where there are currently only two digits, finally, we concatenate the variables to create a single string variable that contains month, day, and four-digit years.
First, input the dates:
input str18 date 20jan2007 16June06 06sept1985 21june04 4july90 9jan1999 6aug99 19august2003 end
Next, we want to identify the day of the month and place it in a variable called day. To do this we instruct Stata to find the day by looking at the beginning of the string (i.e. the date), for one or more values from 0-9. (In other words, look for a number at the start of the line, since we know the first series of numbers is the day.) Generate a new variable day, and set it equal to that value.
gen day = regexs(0) if regexm(date, "^[0-9]+")
The line of syntax below finds the month by looking for one or more letters together in the string. Then, generates the variable month and sets it equal to the month identified in the string.
gen month = regexs(0) if regexm(date, "[a-zA-Z]+")
The year is where things get more complex. Note that the values for assigning centuries are based on my knowledge of my “data.” First of all, we extract all the digits for year. We use the "$" operator to indicate that the search is from the end of the string. We then turn the string variable into a numeric variable using Stata’s function "real". The next action involves dealing with two digit years starting with "0". This corresponds to recent years in the twenty first century. To turn these into four-digit years, we concatenate (using the +) the string identified (the two-digit year) with the string "20". Next we will find the two-digit years 10-99, and concatenate those strings with the string "19". Finally, we create the variable date2 which is our date containing only four-digit years. (We could also use the three variables, day, month, and year to to create a date variable using the Stata date functions.)
gen year = regexs(0) if regexm(date, "[0-9]*$")
replace year = "20"+regexs(0) if regexm(year, "^[0][0-9]$")
replace year = "19"+regexs(0) if regexm(year, "^[1-9][0-9]$")
gen date2 = day+month+year
list
+---------------------------------------------------+
| date day month year date2 |
|---------------------------------------------------|
1. | 20jan2007 20 jan 2007 20jan2007 |
2. | 16June06 16 June 2006 16June2006 |
3. | 06sept1985 06 sept 1985 06sept1985 |
4. | 21june04 21 june 2004 21june2004 |
5. | 4july90 4 july 1990 4july1990 |
|---------------------------------------------------|
6. | 9jan1999 9 jan 1999 9jan1999 |
7. | 6aug99 6 aug 1999 6aug1999 |
8. | 19august2003 19 august 2003 19august2003 |
+---------------------------------------------------+
Regular Expressions
Regular expressions are, in general, a way of searching for and in some cases replacing the occurrence of a pattern within a string based on a set of rules. These rules are defined using a set of operators. The following table shows all of the operators Stata accepts, and explains each one. Note that in Stata, regular expressions will always fall within quotation marks.
[ ] Square brackets indicate that one of the characters inside the brackets should be matched. For example, if I wanted to search for a single letter between f and m, I would type "[f-m]" a-z A range specifies that any value within that range is acceptable. This is case sensitive, so a-z is not the same as A-Z, if either case can be counted as a match, include both a-zA-Z. Numeric values are also acceptable as ranges (e.g. 0-9). . A period matches any character. Allows you to match characters that are usually regular expression operators. For example, if you wanted to match a "[" you would type [ instead of just a single [. * Match zero or more of the characters in preceding expression. For example if I wanted to match a number made up of one or more digits if there is a number, but still want to indicate a match if the rest of the expression fits, I could specify [0-9]* + Match one or more of the characters in the preceding expression. For example if I wanted to match a word containing any combination of letters, I would specify [a-zA-Z]+ ? Match either zero or one of the previous expression. ^ When it appears at the beginning of an expression, a "^" indicates that the following expression should appear at the beginning of the string. $ When it appears at the end of an expression, a "$" indicates that the preceding expression should appear at the end of the string. For example, if I wanted to match a number that was the last thing to appear at the end of a string, I would specify "[0-9]+$" | The logical operator or, indicating that either the expression preceding it or following it qualify as a match. ( ) Creates a subexpression within a larger expression. Useful with the "or" perator (i.e. | ), and when extracting and replacing values. For example, if I wanted to extract a numeric value which I know follows directly after a word or set of letters, I could use the regular expression “[a-zA-Z]+([0-9]+)" this matches the whole expression, but allows you to select the portion in the parentheses (called a substring). Handling substrings is discussed in greater detail below. These expressions can be combined to search for a wide variety of strings.
As mentioned above, there are three types of functions that can be preformed with regular expressions in Stata (if you are creative, you can do any number of other things using these functions, but the basic tools are the built in Stata functions). Stata has separate commands for each of the three types of actions regular expressions can perform:
- regexm – used to find matching strings, evaluates to one if there is a match, and zero otherwise
- regexs – used to return the nth substring within an expression matched by regexm (hence, regexm must always be run before regexs, note that an "if" is evaluated first even though it appears later on the line of syntax).
- regexr – used to replace a matched expression with something else.
Each of these has a slightly different syntax. The line below shows the syntax for regexm, that is, the function that matches your regular expression, where the string may either be a string you type in yourself, a string from a macro, or most commonly, the name of a variable. Regular expression is the regular expression for the string you would like to find, note that it must appear in quotation marks.
regexm(string, "regular expression")For regexs, that is, to recall all or a portion of a string, the syntax is:
regexs(n)Where n is the number assigned to the substring you want to extract. The substrings are actually divided when you run regexm. The entire substring is returned in zero, and each substring is numbered sequentially from 1 to n. For example, regexm(“907-789-3939”, “([0-9]*)-([0-9]*)-([0-9]*)”) returns the following:
Subexpression # String Returned 0 907-789-3939 1 907 2 789 3 3939 Note that in subexpressions 1, 2, and 3, the dashes are dropped, since they are not included in the parentheses that mark the subexpressions.
You can take another look at how this works using the following syntax, which uses the display command to run the function.
display regexm("907-789-3939", "([0-9]*)-([0-9]*)-([0-9]*)") display regexs(0) display regexs(1) display regexs(2) display regexs(3)Because they are functions, the regex commands work within other commands (e.g. generate), but cannot be used on their own (i.e. you cannot start a command in Stata with regexm(…)).
Reference
What are regular expressions and how can I use them in Stata?