We Helped With This R Programming Assignment: Have A Similar One?

## <img src="https://datasciencedegree.wisconsin.edu/wp-content/themes/data-gulp/images/logo.svg" width="300">
# 
# 

# # Assignment 7

# ## Problem 1. Introduction
# 
# The first problem has you write some functions in order to be able to do statistics on arbitrary text.  First, we'll write a function determining the length of each word in a given sentence.  Second, we'll apply that function to some given text.  Third, we'll use it to solve a larger problem: determine properties of each sentence in a Jane Austin book.
# 
# ## Problem 1(a).  Word length
# 
# ? Write a function called ```word_length_list()``` which takes a string and returns a list with the length of each word in the string.  
# 
# For each word, count the number of English, alphanumeric characters.  Words are defined as text separated by spaces. Your function should ignore punctuation.  For example, ```word_length_list("Haven't you eaten 8 oranges today?")``` should return ```[6,3,5,1,7,5]```.  
# 
# * Call or create other functions as necessary to organize your work.
# * Write your own code to do this from first principles.  This means using Python built-in functions for splitting text, checking whether characters are punctuation, etc.  
# * Do *not* use Python packages (like nltk) or code directly copied from online resources (such as regular expressions for splitting text) in order to divide sentences. 

def word_length_list(s):
    # leave only valid characters in string - this version does not work with 
, 
    # so string must be cleaned from 
 before using this function
    s = valid_chars(s)
    counts = []
    # split string into words and count letters of each, add into list.
    for word in s.split(' '):
        counts.append(len(word))
    return counts
    
    
def valid_chars(s):
    # leaves only letters, numbers and spaces in a string
    new_s = ''
    valid_c = ' 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
    for c in s:
        if c in valid_c:
            new_s = new_s + c
    return new_s

# this line will run clean when you have solved the problem.
assert(word_length_list("Haven't you eaten 8 oranges today?")==[6,3,5,1,7,5])
# be sure to restart the kernel and run all cells before committing.  
# this ensures the most recent state is what you think it is.

# ## Problem 1(b).  Application -- "A Mourner."
# 
# The text below is an anonymous essay published in The Boston Gazette and Country Journal on January 8, 1770. 
# 
# >The general Sympathy and Concern for the Murder of the Lad by the base and infamous Richardson on the 22d Instant, will be sufficient Reason for your Notifying the Public that he will be buried from his Father’s House in Frogg Lane, opposite Liberty-Tree, on Monday next, when all the Friends of Liberty may have an Opportunity of paying their last Respects to the Remains of this little Hero and first Martyr to the noble Cause--Whose manly Spirit (after this Accident happened) appear’d in his discreet Answers to his Doctor, his Thanks to the Clergymen who prayed with him, and Parents, and while he underwent the greatest Distress of bodily Pain; and with which he met the King of Terrors.  These Things, together with the several heroic Pieces found in his Pocket, particularly Wolfe’s Summit of human Glory, gives Reason to think he had a martial Genius, and would have made a clever Man.
# 					
# > A Mourner.
# 
# (Source:  Michael Sullivan, _Statistics:  Informed Decisions Using Data_, 4th ed.  p. 188-189.)
# 
# ? Use your function ```word_length_list()``` from 1(a) to find the length of each word in "A Mourner". (Note that your output should end in . . ., 3, 1, 7].)
# 
# ###### Notes
# 
# * check out `%pprint`.  It turns off printing each list element on a separate row

%pprint
mourner = 'The general Sympathy and Concern for the Murder of the Lad by the base and infamous Richardson on the 22d Instant, will be sufficient Reason for your Notifying the Public that he will be buried from his Father’s House in Frogg Lane, opposite Liberty-Tree, on Monday next, when all the Friends of Liberty may have an Opportunity of paying their last Respects to the Remains of this little Hero and first Martyr to the noble Cause--Whose manly Spirit (after this Accident happened) appear’d in his discreet Answers to his Doctor, his Thanks to the Clergymen who prayed with him, and Parents, and while he underwent the greatest Distress of bodily Pain; and with which he met the King of Terrors. These Things, together with the several heroic Pieces found in his Pocket, particularly Wolfe’s Summit of human Glory, gives Reason to think he had a martial Genius, and would have made a clever Man. A Mourner.'
word_length_list(mourner)

# ## Problem 1(c). _Pride and Prejudice_.
# 
# This problem is a bit bigger than parts (a) and (b), and requires some bigger thinking.  You'll have to write loops and possibly list comprehensions to solve it!
# 
# ? Create a function 
# ```collect_statistics``` 
# to count the number of words and mean length of words in each sentence of *Pride and Prejudice*.  We have provided `pride.txt` file in the repo, which is available without restrictions from [Project Gutenberg](https://www.gutenberg.org/).  
# 
# ###### Suggestions
# 
# * You should use the `word_length_list` function from 1(a) in your solution.
# * Create additional functions as necessary to organize your work.
# 
# Before you start programming, I suggest you compute the answers by hand, for the first few sentences of *Pride and Prejudice*.  Be sure you go far enough in the file to encounter a few anomalies!  This will give you a sense of
#   * How to solve the problem using a computer.  Make the computer do what you did!  
#   * What the answer looks like, in that you will know the first few terms in the data set you are constructing.
# 
# ###### Requirements and notes
# 
# 
# * A sentence ends with a period, exclamation point, or question mark. A hyphen, dash, or apostrophe does not end a sentence. Quotation marks do not end a sentence. But also, some periods do not end sentences. For example, Mrs., Mr., Dr., Fr., Jr., St., are all commonly occurring abbreviations that almost never end sentences, and they occur enough in Pride and Prejudice that you need to deal with them or your averages will be impacted significantly. An ellipsis sometimes ends a sentence and sometimes does not, but for this assignment you may assume an ellipsis ends a sentence (but note it does not end 3 sentences!) 
# * Do *not* use Python packages (like nltk) or code directly copied from online resources (such as regular expressions for splitting text) in order to divide sentences. Write your own code to do this from first principles.
# * The mean length of words in the sample sentence from 1(a) ```"Haven't you eaten 8 oranges today?"``` is 4.5.
# 
# ###### Output
# 
# * Include comments to explain the purpose and arguments of each function you create.
# * Save your result as a ```.csv``` file and include it with your submission.

def collect_statistics(filename):
    # read txt
    with open(filename, 'r', encoding="utf8") as myfile:
      s = myfile.read()

    # replace multiple '

' with '.' so we could count them as separate sentences
    s = s.replace('

', '. ')
    # replace 
 with space character
    s = s.replace('
', ' ')
    # replace ? and ! with .
    # for axmaple sentence:
    #   “Do you not want to know who has taken it?” cried his wife impatiently.
    # is splitted into two, as directed in the task by saying:
    #   A sentence ends with a period, exclamation point, or question mark
    s = s.replace('?', '.')
    s = s.replace('!', '.')
    # replace Mrs., Mr., Dr., Fr., Jr., St. with versions withoud periods
    s = s.replace('Mrs. ', 'Mrs ')
    s = s.replace('Mr. ', 'Mr ')
    s = s.replace('Dr. ', 'Dr ')
    s = s.replace('Fr. ', 'Fr ')
    s = s.replace('Jr. ', 'Jr ')
    s = s.replace('St. ', 'St ')

    with open('1c_results.txt', 'w', encoding="utf8") as wfile:
        # write headers into file
        wfile.write('word_count_in_sentence	mean_word_length_in_sentence
')
        # split sentences by .
        for sentence in s.split('.'):
            # get lengths of words in sentence
            word_lengths = word_length_list(sentence)
            # remove zeros from lists
            word_lengths = [x for x in word_lengths if x != 0]
            # if sentence not empty
            if len(word_lengths)>0:
                # print(sentence)
                # print(word_lengths)
                mean_length = sum(word_lengths)/len(word_lengths)
                wfile.write(str(len(word_lengths)))
                wfile.write('	')
                wfile.write(str(mean_length))
                wfile.write('
')
collect_statistics('pride.txt')

# ---
# 
# ## Problem 2.  Introduction 
# 
# <img src="https://upload.wikimedia.org/wikipedia/commons/thumb/d/d5/English_letter_frequency_%28alphabetic%29.svg/600px-English_letter_frequency_%28alphabetic%29.svg.png" width="300">
# 
# Here we will be counting e's in text files, capitalized and uncapitalized, and accented and unaccented.
# 
# ? Your code in 2(b) must include a function called 
# ```
#     count_letter_e(filename, ignore_accents, ignore_case)
#     ```
#     
#   That is, `count_letter_e`:
#   * takes a filename, such as ```"pg1342.txt"```, as input, and 
#   * returns the number of _e_'s as output. 
#   * includes two optional arguments, ```ignore_accents``` and ```ignore_case```.
#     * When ```ignore_accents = True```, your function should count accented characters such as _é_, _ê_, and _è_ as the same as _e_.  
#     * When ```ignore_case=True```, your function should treat uppercase and lowercase _e_ as the same letter.
# * The function ```count_letter_e()``` should return a *single number*, the total number of all characters that are being treated as equivalent to _e_.
# * Create other functions as necessary to organize your work.
# * Include comments which explain the purpose and arguments of each function you create.
# * The files are encoded as [utf-8](https://en.wikipedia.org/wiki/UTF-8). When reading the files, you may have to specify the encoding.

# ## Problem 2(a).  Design a Test Suite
# 
# ? Design a test suite (which will be in this case a set of input text) of at least four sentences that will allow you to quickly verify that all four optional argument possibilities are implemented correctly. Make sure that your test suite contains at least one of each of the 8 possible e's (e, é, ê, è, E, É, Ê, È).
# 
# ? Save your test suite as a set of text files (1 sentence per text file) for use in your ```count_letter_e()``` function, and also include each test sentence in a different markdown cell below. For each sentence, count each type of e by hand and report (in the markdown cell) what the output should be for the four possible combinations of true and false for ```ignore_case``` and ```ignore_accents```.
# 
# *Note that you can complete this portion before you have written a single line of code for your function ```count_letter_e()```.
# 

# Test sentence 1:
#   * Sentence: eee
#   * Count with `ignore_case=True, ignore_accents=True`:3
#   * Count with `ignore_case=True, ignore_accents=False`:3
#   * Count with `ignore_case=False, ignore_accents=True`:3
#   * Count with `ignore_case=False, ignore_accents=False`:3

# Test sentence 2:
# * Sentence: éêè
#   * Count with `ignore_case=True, ignore_accents=True`:3
#   * Count with `ignore_case=True, ignore_accents=False`:0
#   * Count with `ignore_case=False, ignore_accents=True`:3
#   * Count with `ignore_case=False, ignore_accents=False`:0
#  

# Test sentence 3:
# * Sentence: EEE
#   * Count with `ignore_case=True, ignore_accents=True`:3
#   * Count with `ignore_case=True, ignore_accents=False`:3
#   * Count with `ignore_case=False, ignore_accents=True`:0
#   * Count with `ignore_case=False, ignore_accents=False`:0

# Test sentence 4:
#  * Sentence: ÉÊÈ
#   * Count with `ignore_case=True, ignore_accents=True`:3
#   * Count with `ignore_case=True, ignore_accents=False`:0
#   * Count with `ignore_case=False, ignore_accents=True`:0
#   * Count with `ignore_case=False, ignore_accents=False`:0

# ? Describe how you can use your four test sentences to detect problems in your implementation of `count_letter_e()`.  Why do you need four sentences, and not just one, or as many as 16?

# we have 16 function calls, so it is enough to test each category of letters with all ifs
# 

# ## Problem 2(b). Create and Test Your Code
# 
# ? Create the functions described in Problem 2 Introduction, and apply them to your test suite from 2(a).  Do *not* use Python packages or code directly copied from online resources.  Write your own functions from first principles.
# 
# ? Print the results of applying the four combinations of optional arguments of your ```count_letter_e()``` function to your test suite. Verify that the output is correct. (If it isn't, modify your code until your function works correctly on your test suite.)

def count_letter_e(filename, ignore_accents = True, ignore_case = True):
    # make function work with string inputs instead of filename:
    if filename[-4::] == '.txt':
        # read txt
        with open(filename, 'r', encoding="utf8") as myfile:
          s = myfile.read()
    else:
        s = filename
    
    count_e = 0
    
    # ignore_case=False, ignore_accents=False
    if not ignore_accents and not ignore_case:
        count_e = s.count('e')
    # ignore_case=True, ignore_accents=False
    elif not ignore_accents and ignore_case:
        count_e = s.count('e') + s.count('E')
    # ignore_case=False, ignore_accents=True
    elif ignore_accents and not ignore_case:
        count_e = s.count('e') + s.count('é') + s.count('ê') + s.count('è')
    # ignore_case=True, ignore_accents=True
    elif ignore_accents and ignore_case:
        count_e = s.count('e') + s.count('é') + s.count('ê') + s.count('è') + 
                    s.count('E') + s.count('É') + s.count('Ê') + s.count('È')
    return count_e

sentence1 = 'eee'
print(sentence1)
print(count_letter_e(sentence1, True, True))
print(count_letter_e(sentence1, False, True))
print(count_letter_e(sentence1, True, False))
print(count_letter_e(sentence1, False, False))

sentence2 = 'éêè'
print(sentence2)
print(count_letter_e(sentence2, True, True))
print(count_letter_e(sentence2, False, True))
print(count_letter_e(sentence2, True, False))
print(count_letter_e(sentence2, False, False))

sentence3 = 'EEE'
print(sentence3)
print(count_letter_e(sentence3, True, True))
print(count_letter_e(sentence3, False, True))
print(count_letter_e(sentence3, True, False))
print(count_letter_e(sentence3, False, False))

sentence4 = 'ÉÊÈ'
print(sentence4)
print(count_letter_e(sentence4, True, True))
print(count_letter_e(sentence4, False, True))
print(count_letter_e(sentence4, True, False))
print(count_letter_e(sentence4, False, False))
# ## Problem 2(c). Apply Your Code
# 
# ? Apply your code from 2(b) to the two provided `.txt` files for _Pride and Prejudice_  and _L'Enlèvement de la redoute_. 
# ? For each file, print the output of all four combinations of the boolena arguments to `count_letter_e`.
# 

print(count_letter_e('pride.txt', True, True))
print(count_letter_e('pride.txt', True, False))
print(count_letter_e('pride.txt', False, True))
print(count_letter_e('pride.txt', False, False))

print(count_letter_e("l'enlevement.txt", True, True))
print(count_letter_e("l'enlevement.txt", True, False))
print(count_letter_e("l'enlevement.txt", False, True))
print(count_letter_e("l'enlevement.txt", False, False))