Diving headlong into data sets is a part of the lesson for anyone working in data science. Often, this means number-crunching, but what do we do when our data set is primarily text-based? We can use regular expressions. In this tutorial, we’re going to take a closer look at how to use regular expressions (regex) in Python. Show
Regular expressions (regex) are essentially text patterns that you can use to automate searching through and replacing elements within strings of text. This can make cleaning and working with text-based data sets much easier, saving you the trouble of having to search through mountains of text by hand. Regular expressions can be used across a variety of programming languages, and they’ve been around for a very long time! In this tutorial, though, we’ll learning about regular expressions in Python, so basic familiarity with key Python concepts like if-else statements, while and for loops, etc., is required. (If you need a refresher on any of this stuff, our introductory Python courses cover all of the relevant topics interactively, right in your browser!) By the end of the tutorial, you’ll be familiar with how Python regex works, and be able to use the basic patterns and functions in Python’s regex module, (To work through the pandas section of this tutorial, you will need to have the pandas library installed. The easiest way to do this is to download Anaconda and work through this tutorial in a Jupyter notebook. For other options, check out the pandas installation guide.)  Our Task: Analyze Spam EmailsIn this tutorial, we’ll use the Fraudulent Email Corpus from Kaggle. It contains thousands of phishing emails sent between 1998 and 2007. They’re pretty entertaining to read. You can find the full corpus here. But we’ll start by learning basic regex commands using a few emails. If you’d like, you can use our test file as well, or you can try this with the full corpus. Introducing Python’s Regex ModuleFirst, we’ll prepare the data set by opening the test file, setting it to read-only, and reading it. We’ll also assign it to a variable, Notice that we precede the directory path with an Now, suppose we want to find out who the emails are from. We could try raw Python on its own: But that’s not giving us exactly what we want. If you take a look at our test file, we could figure out why and fix it, but instead, let’s use Python’s We’ll start by importing Python’s Here’s how it looks: This is essentially the same length as our raw Python, but that’s because it’s a very simple example. The more you’re trying to do, the more effort Python regex is likely to save you. Before we move on, let’s take a closer look at The Common Python Regex PatternsThe pattern we used with
With these regex patterns in hand, you’ll quickly understand our code above as we go on to explain it. Working with Regex PatternsWe can now explain the use of By adding a It looks like adding dots does acquire the rest of the line for us. But, it’s tedious and we don’t know how many dots to add. This is where the asterisk symbol, Let’s construct a greedy search for Because We might even go further and isolate only the name. Let’s use Notice that we use a backslash next to the first quotation mark. The backslash is a special character used for escaping other special characters. For instance, when we want to use a quotation mark as a string literal instead of a special character, we escape it with a backslash like this: After the first quotation mark is matched, What if we want the email address instead? Looks simple enough, doesn’t it? Only the pattern is different. Let’s walk through it. Here’s how we match just the front part of the email address: Emails always contain an @ symbol, so we start with it. The part of the email before the @ symbol might contain alphanumeric characters, which means Now for the pattern behind the @ symbol: The domain name usually contains alphanumeric characters, periods, and a dash sometimes, so a If we look at the line closely, we see that each email is encapsulated within angle brackets, < and >. Our pattern, Email addresses end with an alphanumeric character, so we cap the pattern with Our full email address pattern thus looks like this: Phew! That was quite a bit to work through. Next, we’ll run through some common Common Python Regex Functions
Let’s look at these one by one before using them to bring some order to our data set. re.search()While Like Because We can also see that printing re.split()Suppose we need a quick way to get the domain name of the email addresses. We could do it with three regex operations, like so: The first line is familiar. We return a list of strings, each containing the contents of the re.sub()Another handy We’ve already seen the tasks on the first and second lines before. On the third line, we apply Regex with PandasNow we have the basics of Python regex in hand. But often for data tasks, we’re not actually using raw Python, we’re using the pandas library. Now let’s take our regex skills to the next level by bringing them into a pandas workflow. Don’t worry if you’ve never used pandas before. We’ll walk through the code every step of the way so you never feel lost. But if you’d like to learn about pandas in more detail, check out our pandas tutorial or the fully interactive course we offer on numpy and pandas. Sorting Emails with Python Regex and PandasOur corpus is a single text file containing thousands of emails (though again, for this tutorial we’re using a much smaller file with just two emails, since printing the results of our regex work on the full corpus would make this post far too long). We’ll use regex and pandas to sort the parts of each email into appropriate categories so that the Corpus can be more easily read or analysed. We’ll sort each email into the following categories:
Each of these categories will become a column in our pandas dataframe (i.e., our table). This will make it easier for us work on and analyze each column individually. We’ll keep working with our small sample, but it’s worth reiterating that regular expressions allow us to write more concise code. Concise code reduces the number of operations our machines have to do, which speeds up our analytical process. Working with our small file of two emails, there’s not much difference, but if you try processing the entire corpus with and without regex, you’ll start to see the advantages! Preparing the ScriptTo start, let’s import the libraries we’ll need and get our file opened again. In addition to We’ve also created an empty list, Now, let’s begin applying regex! Note: we cut off the printout above for the sake of brevity. If you print this on your own machine, it will display everything that’s contained in We use the We know this because we looked into the file before we wrote the script. We didn’t have to peruse the thousands of emails in there. Just the first few, to see what the structure of the data looks like. Whenever possible, it’s good to get your eyes on the actual data before you start working with code, as you’ll often discover useful features like this. We’ve taken a screenshot of what the original text file looks like: Emails start with “From r” The green block is the first email. The blue block is the second email. As we can see, both emails start with One reason we use the Fraudulent Email Corpus in this tutorial is to show that when data is disorganized, unfamiliar, and comes without documentation, we can’t rely solely on code to sort it out. It would require a pair of human eyes. As we’ve just shown, we had to look into the corpus itself to study its structure. Disorganized data like this may require a lot of cleaning up. For instance, even though we count 3,977 emails in this set using the full script we’re about to construct for this tutorial, there are actually more. Some emails actually are not preceded by Notice also that we use Getting Every Name and Address With a For LoopNext, we’ll work with the emails in the In the code above, we use a This is a three-step process. It begins by finding the With Step 1, we find the entire But, data isn’t always straightforward. It can contain surprises. For instance, what if there’s no To avoid errors resulting from missing If you’re working along with this tutorial in your own file, you’ve probably already realized that working with regular expressions gets messy. For instance, these if-else statements are the result of using trial and error on the corpus while writing it. Writing code is an iterative process. It’s worth noting that even if this tutorial is making it seem straightforward, actual practice involves a lot more experimentation. In Step 2, we use a familiar regex pattern from before, We’ll use a different tactic for the name. Each name is bounded by the colon, Now, let’s print out the results of our code to see how they look. Again, we have match objects. Every time we apply Before we do this, recall that if there is no In Step 3A, we use an Then, we simply convert the We do almost exactly the same for Just as we did before, we first check that Then, we use the Let’s check out our results. Perfect. We’ve isolated the email address and the sender’s name. We’ve also added them to the dictionary, which will come into play soon. Now that we’ve found the sender’s email address and name, we do exactly the same set of steps to acquire the recipient’s email address and name for the dictionary. First, we find the the Next, we pre-empt the scenario where If Then, we turn the match objects into strings and add them to the dictionary. Because the structure of the Getting the Date of the EmailNow for the date the email was sent. We acquire the And, just as we do for those two fields, we check that the We’ve printed out The date starts with a number. Hence, we use After that, there’s a space. This is accounted for by The full pattern, Next, we do the same check for a value of If Before we go on, we should note a crucial point. If we use Next up, the subject line of the email. Getting the Email SubjectAs before, we use the same code and code structure to acquire the information we need. We’re becoming more familiar with the use of Python regex now, aren’t we? It’s largely the same code as before, except that we substitute Getting the Body of the EmailThe last item to insert into our dictionary is the body of the email. Separating the header from the body of an email is an awfully complicated task, especially when many of the headers are different in one way or another. Consistency is seldom found in raw unorganised data. Luckily for us, the work’s already been done. Python’s Remember that we’ve already imported the package earlier. Now, we apply its Next, we apply its Why the Email Package and Not Regex for the BodyYou may ask, why use the It’s worth checking out how we arrive at decisions like this one. However, we need to understand what square brackets, Now, we can better understand how we made the decision to use the email package instead. A peek at the data set reveals that email headers stop at the strings Unfortunately, some emails have more than one Create the List of DictionariesFinally, append the dictionary, We might want to print the We could also run Here’s the code in full: And here’s what you’ll get if you run that using our sample text file: We’ve printed out the first item in the We’ve substituted Manipulating Data with PandasWith dictionaries in a list, we’ve made it infinitely easy for the pandas library to do its job. Each key will become a column title, and each value becomes a row in that column. All we have to do is apply the following code: With this single line, we turn the That’s it. We now have a sophisticated pandas dataframe. This is essentially a neat and clean table containing all the information we’ve extracted from the emails. Let’s look at the first few rows. The We can also find precisely what we want. For instance, we can find all the emails sent from a particular domain name. However, let’s learn a new regex pattern to improve our precision in finding the items we want. The pipe symbol, Now, let’s use We’ve used a rather lengthy line of code here. Let’s start from the inside out. We can view emails from individual cells too. To do this, we go through four steps. In Step 1, we find the index of the row where the In Step 2, we use the index to find the email address, which the In Step 3, we extract the email address from the Series object as we would items from a list. You can see that its type is now class. Step 4 is where we extract the email body. In Step 4, As you can see, we can work with regex in many ways, and it plays well with pandas, too! Don’t be discouraged if your regex work includes a lot of trial and error, especially when you’re just getting started! Other ResourcesRegex has grown tremendously since it leapt from biology to engineering all those years ago. Today, regex is used across different programming languages, where there are some variations beyond its basic patterns. We’ve learned a lot of Python regex, and if you’d like to take this to the next level, our Python Data Cleaning Advanced course might be a good fit. You may also find some help in official references, like Python’s documentation for its If you’re so inclined, you can also start exploring the differences between Python regex and other forms of regex Stack Overflow post. Wikipedia has a comparing the different regex engines. If you require data sets to experiment with, Kaggle and are useful. Finally, here’s a Regex cheatsheet we made that is also quite useful. Did this tutorial help?Choose your path to keep learning valuable data skills. Python Tutorials Practice your Python programming skills as you work through our free tutorials. Data science courses Commit to your study with our interactive, in-your-browser data science courses in Python, R, SQL, and more. Data ScienceintermediateLearn PythonPandaspythonregexregular expressionsTutorials About the author Alex YangAlex is a writer fascinated by the things code can do. He also enjoys citizen science and new media art. Modul apa dalam python untuk menjalankan RegEx?Modul RegEx
Python memiliki paket bawaan yang disebut re , yang dapat digunakan untuk bekerja dengan Ekspresi Reguler.
Apa itu python RegEx?Regex merupakan singkatan dari Regular Expression yang merupakan serangkaian karakter yang mendefinisikan sebuah pola pencarian. Beberapa bidang yang menggunakan metode ini adalah seperti Natural Language Processing (NLP), Text Mining, Data Validation, Finding Pattern, Anomaly Detection dan lainnya.
Apa itu string di Python?Mengenal Apa itu String
String dalam bahasa pemrograman Python disebut sebagai kumpulan karakter yang dikelilingi oleh tanda kutip tunggal, tanda kutip ganda bahkan tanda kutip tiga. Komputer tidak memahami karakter. Secara internal, tipe string ini menyimpan karakter yang dimanipulasi sebagai kombinasi dari 0 dan 1.
|
Cara menggunakan python regex translate
Pos Terkait
Copyright © 2024 apacode Inc.
