LESSON LIST

Regular Expressions in Python

Introduction

Regular expressions also known as Regex are used to find search patterns. They are a sequence of characters that can be used to describe what we are trying to search. They can be used for all text based search and replace operations.

Regular Expressions in python

The match Function

The match function searches for a pattern in a string which can be passed with two optional flags. This is the syntax for the match function.

Parameter Description:

Pattern => This is a regular expression that has to be matched.
String => This is the string that has to be searched to match the pattern

The search function

The search functions searches for the first occurence of a pattern in a string that can again be passed with two optional flags. This is the syntax for the search function.

Syntax:

import re
re.search(pattern, string, flags=0)

Parameter Description:

Parameter	Description
pattern	This is the regular expression to be matched.
string	This is the string, which would be searched to match the pattern anywhere in the string.
flags	You can specify different flags using bitwise OR (\|). These are modifiers, which are listed in the table below.

Match Object Methods	Description
group(num=0)	This method returns entire match (or specific subgroup num)
groups()	This method returns all matching subgroups in a tuple (empty if there weren't any)

The re.search returns a match object if successful else returns None.

Example

#!/usr/bin/python
import re

line = "Tigers are smarter than Lions";

searc = re.search( r'(.*) than (.*?) .*', line, re.M|re.I)

if searc:
   print "searc.group() : ", searc.group()
   print "searc.group(1) : ", searc.group(1)
   print "searc.group(2) : ", searc.group(2)
else:
   print "Nothing found!!"

The output will be like

searc.group() :  Tigers are smarter than Lions
searchObj.group(1) :  samrter
searchObj.group(2) :  Lions

The sub function

The sub function searches for the given pattern and replaces the the matching string with the user provided string.

Syntax

import re
re.sub(pattern, repl, string, max=0)

Example

#!/usr/bin/python
import re
phone = "8500-451-999 # This is Phone Number"
# Delete Python-style comments
num = re.sub(r'#.*$', "", phone)
print "Phone Num : ", num
# Remove anything other than digits
num = re.sub(r'\D', "", phone)    
print "Phone Num : ", num

Output

Phone Num :  8500-451-999
Phone Num :  8500451999


    
        '''
python program to find the total occurences of a symbol in string using reqular expressions

Input
enter the main string:
1qaz!@#$!@#$zxswedc@#$%
enter the symbol you wish find occurences:
@

Output
@ occured 3 times in 1qaz!@#$!@#$zxswedc@#$%

'''
import re
msg= 'enter the main string:'
print(msg)
#getting main string from user
k = raw_input()
msg = 'enter the symbol you wish find occurences:'
print(msg)
l = len(k)
#getting symbol from user
s = raw_input()
#replacing input symbol in the string and find the decrease in lenth to find the occurences
#re's replaces all the matching patterns from given input and returns a replaced string
copy = re.sub(r''+s+'','',k)
print(s+' occured '+str(l-len(copy))+' times in '+k)

        Try

The findall function

The findall function provides all the relevant matches of pattern in a string as a list.

Syntax

import re
re.findall(r'pattern',string)

Example

import re
print(re.findall(r'\w','Greycampus'))

Output

['G', 'r', 'e', 'y', 'c', 'a', 'm', 'p', 'u', 's']


    
        '''
Python program to Extract the mail addresses in the given string using regular expression

Input
enter the raw data containing mail addresses:
qwer@qw.comasd@as.comzxcd@sdf.com zxswe@rt1.comjuk9.ju@rt.com

Output
extracted mail addresses are:
qwer@qw.com
asd@as.com
zxcd@sdf.com
zxswe@rt1.com
juk9.ju@rt.com

'''
# importing python regex library
import re
msg = 'enter the raw data containing mail addresses:'
print(msg)
data = raw_input()
#finds all patterns that match 'alphanumeric_with_symbols@alphanumeric.com|net|any 3 letter extension'
m = re.findall(r"[a-zA-Z0-9._$%+-]+.[@].[a-zA-Z0-9]+\..[a-zA-Z]{0,1}.",data)
print('extracted mail addresses are:')
#printing all the mail addresses matching the pattern
for i in range(0,len(m)):
    print(m[i])

        Try

The split function

This function splits the string at the point where the pattern matches with the string.

Syntax

import re
re.split(pattern,string)

Example

import re
print(re.split(r'-','Greycampus-Python'))

Output

['Greycampus', 'Python']

The `start` and `end` functions

These functions return the indices of the start and end respectively of the substring matched by the pattern.

Example

import re
k = re.search(r'\d+','13vv1a1238')
k.end()
k.start()

ouput

2
0

Regular Expression Modifiers: Option Flags

Modifier	Description
re.I	Performs case-insensitive matching.
re.L	Interprets words according to the current locale. This interpretation affects the alphabetic group (\w and \W), as well as word boundary behavior (\b and \B).
re.M	Makes $ match the end of a line (not just the end of the string) and makes ^ match the start of any line (not just the start of the string).
re.S	Makes a period (dot) match any character, including a newline.
re.U	Interprets letters according to the Unicode character set. This flag affects the behavior of \w, \W, \b, \B.
re.X	Permits "cuter" regular expression syntax. It ignores whitespace (except inside a set [] or when escaped by a backslash) and treats unescaped # as a comment marker.

Regular Expression Patterns

Except control characters + ? . * ^ $ ( ) [ ] { } | \ all characters match themselves. You can escape a control character by preceeding it with a backslash.

Pattern	Description
^	Matches beginning of line.
$	Matches end of line.
.	Matches any single character except newline. Using m option allows it to match newline as well.
[...]	Matches any single character in brackets.
[^...]	Matches any single character not in brackets
re*	Matches 0 or more occurrences of preceding expression.
re+	Matches 1 or more occurrence of preceding expression.
re?	Matches 0 or 1 occurrence of preceding expression.
re{ n}	Matches exactly n number of occurrences of preceding expression.
re{ n,}	Matches n or more occurrences of preceding expression.
re{ n, m}	Matches at least n and at most m occurrences of preceding expression.
a\| b	Matches either a or b.
(re)	Groups regular expressions and remembers matched text.
(?imx)	Temporarily toggles on i, m, or x options within a regular expression. If in parentheses, only that area is affected.
(?-imx)	Temporarily toggles off i, m, or x options within a regular expression. If in parentheses, only that area is affected.
(?: re)	Groups regular expressions without remembering matched text.
(?imx: re)	Temporarily toggles on i, m, or x options within parentheses.
(?-imx: re)	Temporarily toggles off i, m, or x options within parentheses.
(?#...)	Comment.
(?= re)	Specifies position using a pattern. Doesn't have a range.
(?! re)	Specifies position using pattern negation. Doesn't have a range.
(?>; re)	Matches independent pattern without backtracking.
\w	Matches word characters.
\W	Matches nonword characters.
\s	Matches whitespace. Equivalent to [\t\n\r\f].
\S	Matches nonwhitespace.
\d	Matches digits. Equivalent to [0-9].
\D	Matches nondigits.
\A	Matches beginning of string.
\Z	Matches end of string. If a newline exists, it matches just before newline.
\z	Matches end of string.
\G	Matches point where last match finished.
\b	Matches word boundaries when outside brackets. Matches backspace (0x08) when inside brackets.
\B	Matches nonword boundaries.
\n, \t, etc.	Matches newlines, carriage returns, tabs, etc.
\1...\9	Matches nth grouped subexpression.
\10	Matches n^th grouped subexpression if it matched already. Otherwise refers to the octal representation of a character code.

Character classes

Example	Description
[Pp]ython	Match "Python" or "python"
rub[ye]	Match "ruby" or "rube"
[aeiou]	Match any one lowercase vowel
[0-9]	Match any digit; same as [0123456789]
[a-z]	Match any lowercase ASCII letter
[A-Z]	Match any uppercase ASCII letter
[a-zA-Z0-9]	Match any of the above
[^aeiou]	Match anything other than a lowercase vowel
[^0-9]	Match anything other than a digit

Special Character classes

Example	Description
.	Match any character except newline
\d	Match a digit: [0-9]
\D	Match a nondigit: [^0-9]
\s	Match a whitespace character: [ \t\r\n\f]
\S	Match nonwhitespace: [^ \t\r\n\f]
\w	Match a single word character: [A-Za-z0-9_]
\W	Match a nonword character: [^A-Za-z0-9_]

Repetition cases

Example	Description
python?	Match "pytho" or "python": the n is optional
ruby*	Match "rub" plus 0 or more ys
python+	Match "pytho" plus 1 or more ns
\d{5}	Match exactly 5 digits
\d{3,}	Match 3 or more digits
\d{6,8}	Match 6, 7, or 8 digits


    
        import re
import tweepy
from tweepy import OAuthHandler
from textblob import TextBlob


global get_tweet_sentiment
global TextBlob
global trim_tweet
global re


DEFAULT_ENCODING = 'utf-8'

def trim_tweet( tweet):
	return ' '.join(re.sub("(@[A-Za-z0-9]+)|([^0-9A-Za-z \t]) | (\w+:\/\/\S+)", " ", tweet).split())

def get_tweet_sentiment( tweet):
  # create TextBlob object of passed tweet text
  analysis = TextBlob(trim_tweet(tweet).decode('ascii','ignore').encode("ascii"))
  # set sentiment
  if analysis.sentiment.polarity > 0:
      return 'positive'
  elif analysis.sentiment.polarity == 0:
      return 'neutral'
  else:
      return 'negative'

def get_tweets(api, query, count = 10):
	# empty list to store parsed tweets
	tweets = []

	try:
		# call twitter api to fetch tweets
		fetched_tweets = api.search(q = query, count = count)

  # parsing tweets one by one
		for tweet in fetched_tweets:
			# empty dictionary to store required params of a tweet
			parsed_tweet = {}
			parsed_tweet['text'] = tweet.text.encode('utf-8')
# saving sentiment of tweet
			parsed_tweet['sentiment'] = get_tweet_sentiment(tweet.text.encode('utf-8'))

			# appending parsed tweet to tweets list
			if tweet.retweet_count > 0:
				# if tweet has retweets, ensure that it is appended only once
				if parsed_tweet not in tweets:
					tweets.append(parsed_tweet)
			else:
				tweets.append(parsed_tweet)

		# return parsed tweets
		return tweets

	except tweepy.TweepError as e:
		# print error (if any)
		print("Error : " + str(e))

# keys and tokens from the Twitter Dev Console
consumer_key = 'DZOgI212lfWF0qTOWdqLnckyk'
consumer_secret = 'b1WgzfpagCLprs89cw3hpHOkDnxUTwprEiAPLfC6hXaugJD9yz'
access_token = '133639112-1xp5SjDByXYdYZdhWU29aijSHjjXozj0ht35KV65'
access_token_secret = 'xGl4IEZ9fALlnPKJqVeVq6WAMYg0h9Koi6Z2TEUCcdgxL'

auth = OAuthHandler(consumer_key, consumer_secret)
# set access token and secret
auth.set_access_token(access_token, access_token_secret)
# create tweepy API object to fetch tweets
api = tweepy.API(auth)
tweets = get_tweets(api,query = 'modi', count = 1000)
ptweets = [tweet for tweet in tweets if tweet['sentiment'] == 'positive']
# percentage of positive tweets
print("Positive tweets percentage: {} %".format(100*len(ptweets)/len(tweets)))
# picking negative tweets from tweets
ntweets = [tweet for tweet in tweets if tweet['sentiment'] == 'negative']
# percentage of negative tweets
print("Negative tweets percentage: {} %".format(100*len(ntweets)/len(tweets)))

# printing first 5 positive tweets
print("\n\nPositive tweets:")
for tweet in ptweets[:5]:
    print(tweet['text'])

# printing first 5 negative tweets
print("\n\nNegative tweets:")
for tweet in ntweets[:5]:
    print(tweet['text'])
                
        Try

Regular Expressions in Python

Introduction

Parameter Description:

The search function

Parameter Description:

Example

The sub function

Example

The findall function

Example

The split function

Example

The start and end functions

Example

Regular Expression Modifiers: Option Flags

Regular Expression Patterns

Character classes

Special Character classes

Repetition cases

The `start` and `end` functions