Aravind's Blog | R | Economics | Finance

rfm 0.2.2

Fri, 31 Jul 2020 00:00:00 +0000

We’re excited to announce the release of rfm 0.2.2 on CRAN! rfm provides tools for customer segmentation using Recency Frequency Monetary value analysis. It includes a Shiny app for interactive segmentation. You can install rfm with:

install.packages("rfm")

In this blog post, we will summarize the changes implemented in the current (0.2.2) and previous release (0.2.1).

Segmentation

In previous versions, rfm_segment() would overwrite a segment if the intervals used to define the segment was a subset of another segment. It was expected that the end user would be careful to ensure that the intervals for each segment would be unique and not a subset of any other segment. You can see the example here.

We are grateful to @leungi for bringing this to our attention and also for fixing it. Now, rfm_segment() does not overwrite the segments even if the intervals for one segment is a subset of another.

# analysis date
analysis_date <- lubridate::as_date("2006-12-31")

# rfm score
rfm_result <- rfm_table_order(rfm_data_orders, customer_id, order_date,
                              revenue, analysis_date)

rfm_result

## # A tibble: 995 x 9
##    customer_id date_most_recent recency_days transaction_cou~ amount
##    <chr>       <date>                  <dbl>            <dbl>  <dbl>
##  1 Abbey O'Re~ 2006-06-09                205                6    472
##  2 Add Senger  2006-08-13                140                3    340
##  3 Aden Lesch~ 2006-06-20                194                4    405
##  4 Admiral Se~ 2006-08-21                132                5    448
##  5 Agness O'K~ 2006-10-02                 90                9    843
##  6 Aileen Bar~ 2006-10-08                 84                9    763
##  7 Ailene Her~ 2006-03-25                281                8    699
##  8 Aiyanna Br~ 2006-04-29                246                4    157
##  9 Ala Schmid~ 2006-01-16                349                3    363
## 10 Alannah Bo~ 2005-04-21                619                4    196
## # ... with 985 more rows, and 4 more variables: recency_score <int>,
## #   frequency_score <int>, monetary_score <int>, rfm_score <dbl>

# segmentation
segment_names <- c(
  "Champions", "Loyal Customers", "Potential Loyalist",
  "New Customers", "Promising", "Need Attention", "About To Sleep",
  "At Risk", "Can't Lose Them", "Lost"
)

recency_lower <- c(4, 2, 3, 4, 3, 2, 2, 1, 1, 1)
recency_upper <- c(5, 5, 5, 5, 4, 3, 3, 2, 1, 2)
frequency_lower <- c(4, 3, 1, 1, 1, 2, 1, 2, 4, 1)
frequency_upper <- c(5, 5, 3, 1, 1, 3, 2, 5, 5, 2)
monetary_lower <- c(4, 3, 1, 1, 1, 2, 1, 2, 4, 1)
monetary_upper <- c(5, 5, 3, 1, 1, 3, 2, 5, 5, 2)

segments <-
  rfm_segment(
    rfm_result,
    segment_names,
    recency_lower,
    recency_upper,
    frequency_lower,
    frequency_upper,
    monetary_lower,
    monetary_upper
  )

# segment size
segments %>%
  count(segment) %>%
  arrange(desc(n)) %>%
  rename(Segment = segment, Count = n)

## # A tibble: 8 x 2
##   Segment            Count
##   <chr>              <int>
## 1 Loyal Customers      278
## 2 Potential Loyalist   229
## 3 Champions            158
## 4 Lost                 111
## 5 At Risk               86
## 6 About To Sleep        50
## 7 Others                48
## 8 Need Attention        35

In the above example, the interval used to define the Champions segment is a subset of Loyal Customers. In the previous versions, those customers who should have been assigned Champions were reassigned as Loyal Customers if the criteria for Champions was evaluated before Loyal Customers. From version 0.2.0, rfm_segment() will avoid such overwriting.

Visualization

rfm used print all the plots by default instead of returning a plot object. This resulted in difficulties for some end users who wanted to:

further modify the plot
include the plot in a panel of other plots

From version 0.2.1, all plotting functions use an additional argument print_plot. It is set to TRUE by default to avoid any disruption to current work flows. Those users who want a plot object to be returned can set the above argument to FALSE.

# analysis date
analysis_date <- lubridate::as_date('2007-01-01')

# transactions data
rfm_order <- rfm_table_order(rfm_data_orders, customer_id, order_date,
                             revenue, analysis_date)

# customer data
rfm_customer <- rfm_table_customer(rfm_data_customer, customer_id,
                                   number_of_orders, recency_days, revenue,
                                   analysis_date)

# plots
p1 <- 
  rfm_heatmap(rfm_order, 
              plot_title = "Transaction Data", 
              print_plot = FALSE)

p2 <- 
  rfm_heatmap(rfm_customer, 
              plot_title = "Customer Data", 
              print_plot = FALSE)

# using patchwork
p1 + p2

Custom Threshold for RFM Scores

Lots of users wanted to know the threshold used for generating the RFM scores. From version 0.2.1, rfm_table_* family of functions return the threshold.

analysis_date <- lubridate::as_date('2006-12-31')
result <- rfm_table_order(rfm_data_orders, customer_id, order_date,
                          revenue, analysis_date)

# threshold
result$threshold

## # A tibble: 5 x 6
##   recency_lower recency_upper frequency_lower frequency_upper monetary_lower
##           <dbl>         <dbl>           <dbl>           <dbl>          <dbl>
## 1            1           115                1               4            12 
## 2          115           181                4               5           256.
## 3          181           297.               5               6           382 
## 4          297.          482                6               8           506.
## 5          482           977                8              15           666 
## # ... with 1 more variable: monetary_upper <dbl>

Another request (see here) was to be able to use custom or user specific threshold for generating RFM score. rfm uses quantiles to generate the lower and upper thresholds used for generating the scores. Unfortunately, if the data is skewed, using quantiles is not effective. From version 0.2.1, users can specify custom threshold for generating the RFM score and we will learn how to do this using an example.

analysis_date <- lubridate::as_date('2006-12-31')
result <- rfm_table_order(rfm_data_orders, customer_id, order_date, revenue,
                          analysis_date)
result$threshold

## # A tibble: 5 x 6
##   recency_lower recency_upper frequency_lower frequency_upper monetary_lower
##           <dbl>         <dbl>           <dbl>           <dbl>          <dbl>
## 1            1           115                1               4            12 
## 2          115           181                4               5           256.
## 3          181           297.               5               6           382 
## 4          297.          482                6               8           506.
## 5          482           977                8              15           666 
## # ... with 1 more variable: monetary_upper <dbl>

If you look at the above output, we have 5 bins/scores and there are six different values. Let us focus on the monetary_* columns in the threshold table. The lower threshold of the first bin and the upper threshold of the last bin are the min and max values form the revenue column of rfm_data_orders and the rest of the values are returned by the quantile() function.

revenue <- 
  rfm_data_orders %>% 
  group_by(customer_id) %>% 
  summarize(total = sum(revenue))

## `summarise()` ungrouping (override with `.groups` argument)

# revenue at customer level
revenue

## # A tibble: 995 x 2
##    customer_id        total
##  * <chr>              <dbl>
##  1 Abbey O'Reilly DVM   472
##  2 Add Senger           340
##  3 Aden Lesch Sr.       405
##  4 Admiral Senger       448
##  5 Agness O'Keefe       843
##  6 Aileen Barton        763
##  7 Ailene Hermann       699
##  8 Aiyanna Bruen PhD    157
##  9 Ala Schmidt DDS      363
## 10 Alannah Borer        196
## # ... with 985 more rows

# min and max
min(revenue$total)

## [1] 12

max(revenue$total)

## [1] 1488

Let us look at the quantiles used for generating the scores.

quantile(revenue$total, probs = seq(0, 1, length.out = 6))

##     0%    20%    40%    60%    80%   100% 
##   12.0  254.8  381.0  505.4  665.0 1488.0

The intervals are created in the below style:

Left-closed, right-open: [ a , b ) = { x ∣ a ≤ x < b }

Since rfm uses left closed intervals to generate the scores, we add 1 to all values except the minimum value. Now, let us recreate the RFM scores using custom threshold instead of quantiles.

rfm_table_order(rfm_data_orders, 
                customer_id, 
                order_date, 
                revenue,
                analysis_date, 
                recency_bins = c(115, 181, 297, 482),
                frequency_bins = c(4, 5, 6, 8), 
                monetary_bins = c(256, 382, 506, 666))

## # A tibble: 995 x 9
##    customer_id date_most_recent recency_days transaction_cou~ amount
##    <chr>       <date>                  <dbl>            <dbl>  <dbl>
##  1 Abbey O'Re~ 2006-06-09                205                6    472
##  2 Add Senger  2006-08-13                140                3    340
##  3 Aden Lesch~ 2006-06-20                194                4    405
##  4 Admiral Se~ 2006-08-21                132                5    448
##  5 Agness O'K~ 2006-10-02                 90                9    843
##  6 Aileen Bar~ 2006-10-08                 84                9    763
##  7 Ailene Her~ 2006-03-25                281                8    699
##  8 Aiyanna Br~ 2006-04-29                246                4    157
##  9 Ala Schmid~ 2006-01-16                349                3    363
## 10 Alannah Bo~ 2005-04-21                619                4    196
## # ... with 985 more rows, and 4 more variables: recency_score <int>,
## #   frequency_score <int>, monetary_score <int>, rfm_score <dbl>

We have used the values from the threshold table to reproduce the earlier result. If you observe carefully, we have specified 4 values while generating 5 bins/scores. Whenever using custom threshold, values supplied should be one less than the number of bins/scores generated as rfm internally computes the min and max values. In general, if you have n bins/scores, you only specify the upper threshold for n - 1 bins/scores.

We have tried our best to explain how to use custom threshold but completely understand that it can be confusing to implement at beginning. If you have any questions about this method, feel free to write to us at support@rsquaredacademy.com and our team will be happy to help you.

Acknowledgements

We are grateful to @leungi, @gfagherazzi and @DavidGarciaEstaun for their inputs.

Learning More

Introducing nse2r

Fri, 17 Jan 2020 00:00:00 +0000

We are excited to announce the nse2r package. NSE (National Stock Exchange) is the leading stock exchange of India, located in the city of Mumbai. While users can manually download data from NSE through a browser, importing this data into R becomes cumbersome. The nse2r R package implements the retrieval of data from NSE and aims to reduce the pre-processing steps needed in analyzing such data.

nse2r is inspired by and a port of the Python package nsetools. The authors and contributors for this R package are not affiliated with NSE and NSE does not offer support for this R package.

With nse2r, you can fetch the following data related to:

stocks
- quote for a given stock
- stock description
- validate stock symbol/ticker
- most actively traded stocks in a month
- 52 week high/low
- top gainers/losers for the last trading session
index
- list of NSE indices
- validate index symbol/ticker
- quote for a given index
futures & options
- top gainers/losers for the last trading session
pre open market data
- nifty
- nifty bank
indices advances & declines

Installation

# Install release version from CRAN
install.packages("nse2r")

# Install development version from GitHub
# install.packages("devtools")
devtools::install_github("rsquaredacademy/nse2r")

Usage

nse2r uses consistent prefix nse_ for easy tab completion.

nse_index_ for index
nse_stock_ for stocks
nse_fo_ for futures and options
nse_preopen_ for preopen data

Preprocessing

nse2r does basic data preprocessing which are listed below:

modify column data types from character to numeric and Date
modify column names
- make them more descriptive
- to snake_case from camelCase

Users can retain the names and format as returned by NSE using the clean_names argument and setting it to FALSE.

Quick Overview

Fetch Indices Quote

nse_index_quote()

## # A tibble: 55 x 4
##    index_name        last_traded_price  change percent_change
##    <chr>                         <dbl>   <dbl>          <dbl>
##  1 NIFTY 50 Pre Open           12195.   -29.2           -0.24
##  2 NIFTY 50                    12170.   -54.7           -0.45
##  3 NIFTY NEXT 50               28617.  -137.            -0.48
##  4 NIFTY100 LIQ 15              3700.   -32.4           -0.87
##  5 NIFTY BANK                  30948.  -133.            -0.43
##  6 INDIA VIX                      15.8    0.44           2.85
##  7 NIFTY 100                   12287.   -56.0           -0.45
##  8 NIFTY 500                    9988.   -38.2           -0.38
##  9 NIFTY MIDCAP 100            17952.   -20.7           -0.12
## 10 NIFTY MIDCAP 50              4941.    -4.25          -0.09
## # ... with 45 more rows

# retain original column names as returned by NSE
nse_index_quote(clean_names = FALSE)

## # A tibble: 55 x 4
##    name              lastPrice  change pChange
##    <chr>                 <dbl>   <dbl>   <dbl>
##  1 NIFTY 50 Pre Open   12195.   -29.2    -0.24
##  2 NIFTY 50            12170.   -54.7    -0.45
##  3 NIFTY NEXT 50       28617.  -137.     -0.48
##  4 NIFTY100 LIQ 15      3700.   -32.4    -0.87
##  5 NIFTY BANK          30948.  -133.     -0.43
##  6 INDIA VIX              15.8    0.44    2.85
##  7 NIFTY 100           12287.   -56.0    -0.45
##  8 NIFTY 500            9988.   -38.2    -0.38
##  9 NIFTY MIDCAP 100    17952.   -20.7    -0.12
## 10 NIFTY MIDCAP 50      4941.    -4.25   -0.09
## # ... with 45 more rows

Top gainers for the last trading session.

nse_stock_top_gainers()

## # A tibble: 10 x 12
##    symbol series last_corp_annou~ last_corp_annou~ open_price high_price
##    <chr>  <chr>  <date>           <chr>                 <dbl>      <dbl>
##  1 INFRA~ EQ     2019-12-19       Interim Dividen~       224.       244.
##  2 ZEEL   EQ     2019-07-15       Annual General ~       269        287.
##  3 BPCL   EQ     2019-08-21       Dividend - Rs 8~       458.       466.
##  4 COALI~ EQ     2019-08-09       Annual General ~       201.       205.
##  5 KOTAK~ EQ     2019-07-12       Annual General ~      1605       1631.
##  6 HDFC   EQ     2019-07-19       Annual General ~      2443       2486.
##  7 ULTRA~ EQ     2019-07-10       Annual General ~      4455       4520 
##  8 BRITA~ EQ     2019-08-22       Scheme Of Arang~      3100       3130 
##  9 RELIA~ EQ     2019-08-02       Annual General ~      1529.      1546.
## 10 TECHM  EQ     2019-07-25       Annual General ~       777.       783.
## # ... with 6 more variables: low_price <dbl>, last_traded_price <dbl>,
## #   prev_close_price <dbl>, percent_change <dbl>, traded_quantity <dbl>,
## #   turnover_in_lakhs <dbl>

# retain original column names as returned by NSE
nse_stock_top_gainers(clean_names = FALSE)

## # A tibble: 10 x 12
##    symbol series lastCorpAnnounc~ lastCorpAnnounc~ openPrice highPrice
##    <chr>  <chr>  <date>           <chr>                <dbl>     <dbl>
##  1 INFRA~ EQ     2019-12-19       Interim Dividen~      224.      244.
##  2 ZEEL   EQ     2019-07-15       Annual General ~      269       287.
##  3 BPCL   EQ     2019-08-21       Dividend - Rs 8~      458.      466.
##  4 COALI~ EQ     2019-08-09       Annual General ~      201.      205.
##  5 KOTAK~ EQ     2019-07-12       Annual General ~     1605      1631.
##  6 HDFC   EQ     2019-07-19       Annual General ~     2443      2486.
##  7 ULTRA~ EQ     2019-07-10       Annual General ~     4455      4520 
##  8 BRITA~ EQ     2019-08-22       Scheme Of Arang~     3100      3130 
##  9 RELIA~ EQ     2019-08-02       Annual General ~     1529.     1546.
## 10 TECHM  EQ     2019-07-25       Annual General ~      777.      783.
## # ... with 6 more variables: lowPrice <dbl>, ltp <dbl>,
## #   previousPrice <dbl>, netPrice <dbl>, tradedQuantity <dbl>,
## #   turnoverInLakhs <dbl>

Stocks that have touched their 52 week highs during the day

nse_stock_year_high()

## # A tibble: 44 x 10
##    symbol symbol_desc date       new_high  year last_traded_pri~ prev_high
##    <chr>  <chr>       <date>        <dbl> <dbl>            <dbl>     <dbl>
##  1 AGCNET AGC Networ~ 2020-01-20     223.  223.             223.      212.
##  2 ALKYL~ Alkyl Amin~ 2020-01-17    1300  1300             1295      1300 
##  3 APLAP~ APL Apollo~ 2020-01-15    2022. 2022.            1993.     1977.
##  4 AUBANK AU Small F~ 2020-01-20     892   892              890.      892 
##  5 BANSW~ Banswara S~ 2020-01-20     123   123              118.      123 
##  6 BHART~ Bharti Air~ 2020-01-20     515   515              508.      513.
##  7 CANFI~ Can Fin Ho~ 2019-11-05     467.  467.             467.      434.
##  8 CDSL   Central De~ 2020-01-20     284.  284.             281.      282 
##  9 FINEO~ Fine Organ~ 2020-01-20    2266  2266             2200      2238.
## 10 FORCE~ FORCE MOTO~ 2020-01-20    1504  1504             1470.     1487 
## # ... with 34 more rows, and 3 more variables: prev_close <dbl>,
## #   change <dbl>, percent_change <dbl>

# retain original column names as returned by NSE
nse_stock_year_high(clean_names = FALSE)

## # A tibble: 44 x 10
##    symbol symbolDesc dt         value  year   ltp value_old  prev change
##    <chr>  <chr>      <date>     <dbl> <dbl> <dbl>     <dbl> <dbl>  <dbl>
##  1 AGCNET AGC Netwo~ 2020-01-20  223.  223.  223.      212.  212.  10.6 
##  2 ALKYL~ Alkyl Ami~ 2020-01-17 1300  1300  1295      1300  1272.  22.5 
##  3 APLAP~ APL Apoll~ 2020-01-15 2022. 2022. 1993.     1977. 1943.  50   
##  4 AUBANK AU Small ~ 2020-01-20  892   892   890.      892   886.   4.05
##  5 BANSW~ Banswara ~ 2020-01-20  123   123   118.      123   121.  -3.2 
##  6 BHART~ Bharti Ai~ 2020-01-20  515   515   508.      513.  509.  -0.35
##  7 CANFI~ Can Fin H~ 2019-11-05  467.  467.  467.      434.  389.  77.8 
##  8 CDSL   Central D~ 2020-01-20  284.  284.  281.      282   273.   7.85
##  9 FINEO~ Fine Orga~ 2020-01-20 2266  2266  2200      2238. 2220. -20.3 
## 10 FORCE~ FORCE MOT~ 2020-01-20 1504  1504  1470.     1487  1466.   4.2 
## # ... with 34 more rows, and 1 more variable: pChange <dbl>

Learning More

Feedback

All feedback is welcome. Issues (bugs and feature requests) can be posted to github tracker. For help with code or other related questions, feel free to reach out to us at pkgs@rsquaredacademy.com.

pkginfo: Tools for Retrieving R Package Information

Fri, 05 Jul 2019 00:00:00 +0000

Motivation

There are several wonderful tools for retrieving information about R packages, some of which are listed below:

cranlogs, dlstats and packageRank for R package download stats
pkgsearch and packagefinder for searching CRAN R packages
crandb provides API for programatically accessing meta-data
cchecks for CRAN check results

We have used some or all of these to track/monitor our own R packages available on CRAN. Over time, we wanted to have a single interface which would retrieve information from different places including:

check results from CRAN
issues & pull requests from GitHub
build status from Travis and Appveyor
code coverage from Codecov
questions tagged with the package name in Stack Overflow

And below is a demo of what we came up with…

The above shiny app can be launched with:

pkginfo::pkginfo_dashboard()

Introduction

pkginfo will allow you to retrieve information from multiple places. The shiny app is especially useful if you are looking for information about multiple R packages. pkginfo provides two APIs, one is object oriented, the other one is procedural, and in this post, we will show you how to use them.

Installation

You can install the development version from GitHub with:

devtools::install_github("rsquaredacademy/pkginfo")

Object Oriented API

The object oriented API uses R6 classes.

library(pkginfo)

GitHub

Creating a new GitHubRepo object

myRepo <- GitHubRepo$new("olsrr", "rsquaredacademy")

Stars, Forks & Issues

get_stats() will report the number of stars, forks and open issues.

myRepo$get_stats()

## # A tibble: 1 x 3
##   stars issues forks
##   <int>  <int> <int>
## 1    77     26    11

Issues

get_issues() will report the date, number, author and title of all the open issues.

myRepo$get_issues()

## # A tibble: 26 x 4
##    date       number author        title                                   
##    <date>      <int> <chr>         <chr>                                   
##  1 2019-06-20    133 aravindhebba~ Cook's distance chart threshold         
##  2 2018-12-27    129 aravindhebba~ Bonferroni outlier test                 
##  3 2018-12-24    126 aravindhebba~ Heteroskedasticity corrected covariance~
##  4 2018-12-24    125 aravindhebba~ Test linear hypothesis                  
##  5 2018-12-24    124 aravindhebba~ Ceres plot                              
##  6 2018-12-22    123 aravindhebba~ Power transformations                   
##  7 2018-12-22    122 aravindhebba~ Box Tidwell transformation              
##  8 2018-12-22    121 aravindhebba~ Box Cox transformation                  
##  9 2018-12-21    120 aravindhebba~ Wald test                               
## 10 2018-12-21    119 aravindhebba~ Reset test                              
## # ... with 16 more rows

Branches

get_branches() will report the name of the branches.

myRepo$get_branches()

## # A tibble: 2 x 1
##   branches
##   <chr>   
## 1 develop 
## 2 master

Build Status & Code Coverage

The build status is reported from Travis CI and Appveyor while code coverage is reported from Codecov

Travis CI

myRepo$get_travis_status()

## [1] "Failure"

Codecov

myRepo$get_coverage()

## [1] "82.83485"

Procedural API

GitHub

To use the procedural API for retrieving GitHub information, you need to provide the following inputs:

package/repo name
GitHUb user/organization name (optional)

If you do not know the GitHub user/org name, leave it empty. pkginfo will look for the user/org name in the URLS available on the CRAN page of the package.

Stars, Forks & Issues

get_gh_stats("olsrr", "rsquaredacademy")

## # A tibble: 1 x 3
##   stars issues forks
##   <int>  <int> <int>
## 1    77     26    11

Issues

get_gh_issues("olsrr", "rsquaredacademy")

## # A tibble: 26 x 4
##    date       number author        title                                   
##    <date>      <int> <chr>         <chr>                                   
##  1 2019-06-20    133 aravindhebba~ Cook's distance chart threshold         
##  2 2018-12-27    129 aravindhebba~ Bonferroni outlier test                 
##  3 2018-12-24    126 aravindhebba~ Heteroskedasticity corrected covariance~
##  4 2018-12-24    125 aravindhebba~ Test linear hypothesis                  
##  5 2018-12-24    124 aravindhebba~ Ceres plot                              
##  6 2018-12-22    123 aravindhebba~ Power transformations                   
##  7 2018-12-22    122 aravindhebba~ Box Tidwell transformation              
##  8 2018-12-22    121 aravindhebba~ Box Cox transformation                  
##  9 2018-12-21    120 aravindhebba~ Wald test                               
## 10 2018-12-21    119 aravindhebba~ Reset test                              
## # ... with 16 more rows

CRAN Check Results

The CRAN check results is reported using the cchecksapi API.

get_pkg_cran_check_results("olsrr")

## # A tibble: 12 x 4
##    os               r      status url                                      
##    <chr>            <chr>  <chr>  <chr>                                    
##  1 linux-x86_64-de~ devel  OK     https://www.R-project.org/nosvn/R.check/~
##  2 linux-x86_64-de~ devel  OK     https://www.R-project.org/nosvn/R.check/~
##  3 linux-x86_64-fe~ devel  OK     https://www.R-project.org/nosvn/R.check/~
##  4 linux-x86_64-fe~ devel  OK     https://www.R-project.org/nosvn/R.check/~
##  5 windows-ix86+x8~ devel  OK     https://www.R-project.org/nosvn/R.check/~
##  6 linux-x86_64     patch~ OK     https://www.R-project.org/nosvn/R.check/~
##  7 solaris-x86      patch~ OK     https://www.R-project.org/nosvn/R.check/~
##  8 linux-x86_64     relea~ OK     https://www.R-project.org/nosvn/R.check/~
##  9 windows-ix86+x8~ relea~ OK     https://www.R-project.org/nosvn/R.check/~
## 10 osx-x86_64       relea~ OK     https://www.R-project.org/nosvn/R.check/~
## 11 windows-ix86+x8~ oldrel OK     https://www.R-project.org/nosvn/R.check/~
## 12 osx-x86_64       oldrel OK     https://www.R-project.org/nosvn/R.check/~

Build Status

You can check the build status of a package from both Travis CI and Appveyor.

Travis CI

get_status_travis("olsrr", "rsquaredacademy")

## [1] "Failure"

Appveyor

get_status_appveyor("olsrr", "rsquaredacademy")

## [1] "failed"

Code Coverage

Report code coverage for the package from the coverage service Codecov. We hope to add coverage from Coveralls in the near future.

get_code_coverage("olsrr", "rsquaredacademy")

## [1] "82.83485"

Stack Overflow

Questions tagged with the package name will be displayed from Stack Overflow.

get_so_questions("dplyr")

## # A tibble: 30 x 6
##    date       title               owner   answered views link              
##    <date>     <chr>               <chr>   <lgl>    <int> <chr>             
##  1 2019-07-22 subset data based ~ Amer    TRUE        29 https://stackover~
##  2 2019-07-21 ID variable change~ user88~ FALSE       21 https://stackover~
##  3 2019-07-18 How to group by an~ kdkirc~ FALSE       26 https://stackover~
##  4 2019-07-21 How to do countifs~ frida ~ TRUE        30 https://stackover~
##  5 2019-07-21 sample_frac functi~ squeez~ FALSE       30 https://stackover~
##  6 2019-07-21 Using clusrank by ~ Jason ~ TRUE        29 https://stackover~
##  7 2019-07-21 Fastest way to exp~ JaimieC FALSE       32 https://stackover~
##  8 2019-07-21 What does the !! o~ maleta  FALSE       34 https://stackover~
##  9 2019-07-21 Create an event in~ Marco   TRUE        37 https://stackover~
## 10 2019-07-21 lubridate with_tz ~ hjw     FALSE       23 https://stackover~
## # ... with 20 more rows

To Do..

In the near future, pkginfo will retrieve information:

about packages that are available on GitHub/GitLab but not on CRAN
packages that are available on GitLab
about code coverage from Coveralls
about Twitter mentions

We did not have any specific end user in mind while developing but believe it will be more useful to package developers/maintainers. Suggestions/feedbacks and pull requests are always welcome.

If you see mistakes or want to suggest changes, please create an issue on the source repository or reach out to us at feedback@rsquaredacademy.com.

Shiny Apps for Interactive Data Analysis

Mon, 01 Apr 2019 00:00:00 +0000

I am excited and happy to share a set of shiny apps built for interactive data analysis and teaching at Rsquared Academy. The apps are part of our R packages and presently cover the following topics:

Descriptive Statistics
Probability Distributions
Hypothesis Testing
Linear Regression
Logistic Regression
RFM Analysis
Data Visualization

Demo

Details

You can try the live version of the apps here or locally launch these apps from the R packages mentioned in the below table. It will be easier to use the apps if you refer to the documentation of the respective packages.

Topic	Try App	R Package	API
Descriptive Statistics	Launch	descritpr	`ds_launch_shiny_app()`
Visualize Distributions	Launch	vistributions	`vdist_launch_app()`
Hypothesis Testing	Launch	inferr	`infer_launch_shiny_app()`
Linear Regression	Launch	olsrr	`ols_launch_app()`
Logistic Regression	Launch	blorr	`blr_launch_app()`
RFM Analysis	Launch	rfm	`rfm_launch_app()`
Data Visualization	Launch	xplorerr	`app_visualizer()`

Feedback

All feedback is welcome. Issues (bugs and feature requests) can be posted to github tracker. For help with code or other related questions, feel free to reach out to us at pkgs@rsquaredacademy.com.

Visually explore Probability Distributions with vistributions

Thu, 14 Mar 2019 00:00:00 +0000

I am happy to introduce the vistributions package, a set of tools for visually exploring probability distributions.

Installation

# Install release version from CRAN
install.packages("vistributions")

# Install development version from GitHub
# install.packages("devtools")
devtools::install_github("rsquaredacademy/vistributions")

Shiny App

vistributions includes a shiny app which can be launched using

vdist_launch_app()

or try the live version here.

Read on to learn more about the features of vistributions, or see the vistributions website for detailed documentation on using the package.

Tab Completion

The common vdist_ prefix will trigger autocomplete, allowing you to see all vistributions functions:

In exploring statistical distributions, we focus on the following:

what influences the shape of a distribution
calculate probability from a given quantile
calculate quantiles out of given probability

for the following distributions:

Normal
Binomial
Chi Square
F
t

Normal Distribution

Distribution Shape

Visualize how changes in mean and standard deviation affect the shape of the normal distribution.

Input

mean: mean of the normal distribution
sd: standard deviation of the normal distribution

vdist_normal_plot(mean = 2, sd = 0.6)

Percentiles

Calculate and visualize quantiles out of given probability.

Input

probs: a probability value
mean: mean of the normal distribution
sd: standard deviation of the normal distribution
type: lower/upper tail

Suppose X, the grade on a exam, is normally distributed with mean 60 and standard deviation 3. The teacher wants to give 10% of the class an A. What should be the cutoff to determine who gets an A?

vdist_normal_perc(0.10, 60, 3, 'upper')

The teacher wants to give lower 15% of the class a D. What cutoff should the teacher use to determine who gets an D?

vdist_normal_perc(0.85, 60, 3, 'lower')

The teacher wants to give middle 50% of the class a B. What cutoff should the teacher use to determine who gets an B?

vdist_normal_perc(0.5, 60, 3, 'both')

Probabilities

Calculate and visualize probability from a given quantile

Input

perc: a quantile value
mean: mean of the normal distribution
sd: standard deviation of the normal distribution
type: lower/upper/both tail

Let X be the IQ of a randomly selected student of a school. Assume X ~ N(90, 4). What is the probability that a randomly selected student has an IQ below 80?

vdist_normal_prob(80, mean = 90, sd = 4)

What is the probability that a randomly selected student has an IQ above 100?

vdist_normal_prob(100, mean = 90, sd = 4, type = 'upper')

What is the probability that a randomly selected student has an IQ between 85 and 100?

vdist_normal_prob(c(85, 100), mean = 90, sd = 4, type = 'both')

Binomial Distribution

Distribution Shape

Visualize how changes in number of trials and the probability of success affect the shape of the binomial distribution.

vdist_binom_plot(10, 0.3)

Percentiles

Calculate and visualize quantiles out of given probability

Input

p: a single aggregated probability of multiple trials
n: the number of trials
tp: the probability of success in a trial
type: lower/upper tail

vdist_binom_perc(10, 0.5, 0.05)

vdist_binom_perc(10, 0.5, 0.05, 'upper')

Probabilities

Calculate and visualize probability from a given quantile

Input

p: probability of success
n: the number of trials
s: number of success in a trial
type: lower/upper/interval/exact tail

Assume twenty-percent (20%) of Magemill have no health insurance. Randomly sample n = 12 Magemillians. Let X denote the number in the sample with no health insurance. What is the probability that exactly 4 of the 15 sampled have no health insurance?

vdist_binom_prob(12, 0.2, 4, type = 'exact')

What is the probability that at most one of those sampled has no health insurance?

vdist_binom_prob(12, 0.2, 1, 'lower')

What is the probability that more than seven have no health insurance?

vdist_binom_prob(12, 0.2, 8, 'upper')

What is the probability that fewer than 5 have no health insurance?

vdist_binom_prob(12, 0.2, c(0, 4), 'interval')

Chi Square Distribution

Distribution Shape

Visualize how changes in degrees of freedom affect the shape of the chi square distribution.

vdist_chisquare_plot(df = 5)

vdist_chisquare_plot(df = 5, normal = TRUE)

Percentiles

Calculate quantiles out of given probability

Input

probs: a probability value
df: degrees of freedom
type: lower/upper tail

Let X be a chi-square random variable with 8 degrees of freedom. What is the upper fifth percentile?

vdist_chisquare_perc(0.05, 8, 'upper')

What is the tenth percentile?

vdist_chisquare_perc(0.10, 8, 'lower')

Probability

Calculate probability from a given quantile.

Input

perc: a quantile value
df: degrees of freedom
type: lower/upper tail

What is the probability that a chi-square random variable with 12 degrees of freedom is greater than 8.79?

vdist_chisquare_prob(8.79, 12, 'upper')

What is the probability that a chi-square random variable with 12 degrees of freedom is greater than 8.62?

vdist_chisquare_prob(8.62, 12, 'lower')

F Distribution

Distribution Shape

Visualize how changes in degrees of freedom affect the shape of the F distribution.

vdist_f_plot()

vdist_f_plot(6, 10, normal = TRUE)

Percentiles

Calculate quantiles out of given probability

Input

probs: a probability value
num_df: nmerator degrees of freedom
den_df: denominator degrees of freedom
type: lower/upper tail

Let X be an F random variable with 4 numerator degrees of freedom and 5 denominator degrees of freedom. What is the upper twenth percentile?

vdist_f_perc(0.20, 4, 5, 'upper')

What is the 35th percentile?

vdist_f_perc(0.35, 4, 5, 'lower')

Probabilities

Calculate probability from a given quantile.

Input

perc: a quantile value
num_df: nmerator degrees of freedom
den_df: denominator degrees of freedom
type: lower/upper tail

What is the probability that an F random variable with 4 numerator degrees of freedom and 5 denominator degrees of freedom is greater than 3.89?

vdist_f_prob(3.89, 4, 5, 'upper')

What is the probability that an F random variable with 4 numerator degrees of freedom and 5 denominator degrees of freedom is less than 2.63?

vdist_f_prob(2.63, 4, 5, 'lower')

t Distribution

Distribution Shape

Visualize how degrees of freedom affect the shape of t distribution.

vdist_t_plot(df = 8)

Percentiles

Calculate quantiles out of given probability

Input

probs: a probability value
df: degrees of freedom
type: lower/upper/both tail

What is the upper fifteenth percentile?

vdist_t_perc(0.15, 8, 'upper')

What is the eleventh percentile?

vdist_t_perc(0.11, 8, 'lower')

What is the area of the curve that has 95% of the t values?

vdist_t_perc(0.8, 8, 'both')

Probabilities

Calculate probability from a given quantile.

Input

perc: a quantile value
df: degrees of freedom
type: lower/upper/interval/both tail

Let T follow a t-distribution with r = 6 df.

What is the probability that the value of T is less than 2?

vdist_t_prob(2, 6, 'lower')

What is the probability that the value of T is greater than 2?

vdist_t_prob(2, 6, 'upper')

What is the probability that the value of T is between -2 and 2?

vdist_t_prob(2, 6, 'both')

What is the probability that the absolute value of T is greater than 2?

vdist_t_prob(2, 6, 'interval')

Learning More

The vistributions website includes comprehensive documentation on using the package, including the following article that gives a brief introduction to vistributions:

Introduction to vistributions

Feedback

vistributions has been on CRAN for a few months now while we were fixing bugs and making the API stable. All feedback is welcome. Issues (bugs and feature requests) can be posted to github tracker. For help with code or other related questions, feel free to reach out to us at pkgs@rsquaredacademy.com.

Binning Data with rbin

Tue, 12 Mar 2019 00:00:00 +0000

I am happy to introduce the rbin package, a set of tools for binning/discretization of data, designed keeping in mind beginner/intermediate R users. It includes two RStudio addins for interactive binning.

Installation

# Install release version from CRAN
install.packages("rbin")

# Install development version from GitHub
# install.packages("devtools")
devtools::install_github("rsquaredacademy/rbin")

RStudio Addins

rbin includes two RStudio addins for manually binning data. Below is a demo:

Read on to learn more about the features of rbin, or see the rbin website for detailed documentation on using the package.

Introduction

Binning is the process of transforming numerical or continuous data into categorical data. It is a common data pre-processing step of the model building process. rbin has the following features:

manual binning using shiny app
equal length binning method
winsorized binning method
quantile binning method
combine levels of categorical data
create dummy variables based on binning method
calculates weight of evidence (WOE), entropy and information value (IV)
provides summary information about binning process

Manual Binning

For manual binning, you need to specify the cut points for the bins. rbin follows the left closed and right open interval ([0,1) = {x | 0 ≤ x < 1}) for creating bins. The number of cut points you specify is one less than the number of bins you want to create i.e. if you want to create 10 bins, you need to specify only 9 cut points as shown in the below example. The accompanying RStudio addin, rbinAddin() can be used to iteratively bin the data and to enforce monotonic increasing/decreasing trend.

After finalizing the bins, you can use rbin_create() to create the dummy variables.

Bins

bins <- rbin_manual(mbank, y, age, c(29, 31, 34, 36, 39, 42, 46, 51, 56))
bins

## Binning Summary
## ---------------------------
## Method               Manual 
## Response             y 
## Predictor            age 
## Bins                 10 
## Count                4521 
## Goods                517 
## Bads                 4004 
## Entropy              0.5 
## Information Value    0.12 
## 
## 
## # A tibble: 10 x 7
##    cut_point bin_count  good   bad      woe         iv entropy
##    <chr>         <int> <int> <int>    <dbl>      <dbl>   <dbl>
##  1 < 29            410    71   339 -0.484   0.0255       0.665
##  2 < 31            313    41   272 -0.155   0.00176      0.560
##  3 < 34            567    55   512  0.184   0.00395      0.459
##  4 < 36            396    45   351  0.00712 0.00000443   0.511
##  5 < 39            519    47   472  0.260   0.00701      0.438
##  6 < 42            431    33   398  0.443   0.0158       0.390
##  7 < 46            449    47   402  0.0993  0.000942     0.484
##  8 < 51            521    40   481  0.440   0.0188       0.391
##  9 < 56            445    49   396  0.0426  0.000176     0.500
## 10 >= 56           470    89   381 -0.593   0.0456       0.700

Plot

plot(bins)

Dummy Variables

bins <- rbin_manual(mbank, y, age, c(29, 31, 34, 36, 39, 42, 46, 51, 56))
rbin_create(mbank, age, bins)

## # A tibble: 4,521 x 26
##      age job           marital education default balance housing loan 
##    <int> <fct>         <fct>   <fct>     <fct>     <dbl> <fct>   <fct>
##  1    34 technician    married tertiary  no          297 yes     no   
##  2    49 services      married secondary no          180 yes     yes  
##  3    38 admin.        single  secondary no          262 no      no   
##  4    47 services      married secondary no          367 yes     no   
##  5    51 self-employed single  secondary no         1640 yes     no   
##  6    40 unemployed    married secondary no         3382 yes     no   
##  7    58 retired       married secondary no         1227 no      no   
##  8    32 unemployed    married primary   no          309 yes     no   
##  9    46 blue-collar   married secondary no          922 yes     no   
## 10    32 services      married tertiary  no            0 no      no   
##    contact     day month duration campaign pdays previous poutcome y    
##    <fct>     <int> <fct>    <int>    <int> <int>    <int> <fct>    <fct>
##  1 cellular     29 jan        375        2    -1        0 unknown  0    
##  2 unknown       2 jun        392        3    -1        0 unknown  0    
##  3 cellular      3 feb        315        2   180        6 failure  1    
##  4 cellular     12 may        309        1   306        4 success  1    
##  5 unknown      15 may         67        4    -1        0 unknown  0    
##  6 unknown      14 may        125        1    -1        0 unknown  0    
##  7 cellular     14 aug        182        2    37        2 failure  0    
##  8 telephone    13 may        185        1   370        3 failure  0    
##  9 telephone    18 nov        296        2    -1        0 unknown  0    
## 10 cellular     21 nov         80        1    -1        0 unknown  0    
##    `age_<_31` `age_<_34` `age_<_36` `age_<_39` `age_<_42` `age_<_46`
##         <dbl>      <dbl>      <dbl>      <dbl>      <dbl>      <dbl>
##  1          0          0          1          0          0          0
##  2          0          0          0          0          0          0
##  3          0          0          0          1          0          0
##  4          0          0          0          0          0          0
##  5          0          0          0          0          0          0
##  6          0          0          0          0          1          0
##  7          0          0          0          0          0          0
##  8          0          1          0          0          0          0
##  9          0          0          0          0          0          0
## 10          0          1          0          0          0          0
##    `age_<_51` `age_<_56` `age_>=_56`
##         <dbl>      <dbl>       <dbl>
##  1          0          0           0
##  2          1          0           0
##  3          0          0           0
##  4          1          0           0
##  5          0          1           0
##  6          0          0           0
##  7          0          0           1
##  8          0          0           0
##  9          1          0           0
## 10          0          0           0
## # ... with 4,511 more rows

Factor Binning

You can collapse or combine levels of a factor/categorical variable using rbin_factor_combine() and then use rbin_factor() to look at weight of evidence, entropy and information value. After finalizing the bins, you can use rbin_factor_create() to create the dummy variables. You can use the RStudio addin, rbinFactorAddin() to interactively combine the levels and create dummy variables after finalizing the bins.

Combine Levels

upper <- c("secondary", "tertiary")
out <- rbin_factor_combine(mbank, education, upper, "upper")
table(out$education)

## 
## primary unknown   upper 
##     691     179    3651

out <- rbin_factor_combine(mbank, education, c("secondary", "tertiary"), "upper")
table(out$education)

## 
## primary unknown   upper 
##     691     179    3651

Bins

bins <- rbin_factor(mbank, y, education)
bins

## Binning Summary
## ---------------------------
## Method               Custom 
## Response             y 
## Predictor            education 
## Levels               4 
## Count                4521 
## Goods                517 
## Bads                 4004 
## Entropy              0.51 
## Information Value    0.05 
## 
## 
## # A tibble: 4 x 7
##   level     bin_count  good   bad    woe      iv entropy
##   <fct>         <int> <int> <int>  <dbl>   <dbl>   <dbl>
## 1 tertiary       1299   195  1104 -0.313 0.0318    0.610
## 2 secondary      2352   231  2121  0.170 0.0141    0.463
## 3 unknown         179    25   154 -0.229 0.00227   0.583
## 4 primary         691    66   625  0.201 0.00572   0.455

Plot

plot(bins)

Create Bins

upper <- c("secondary", "tertiary")
out <- rbin_factor_combine(mbank, education, upper, "upper")
rbin_factor_create(out, education)

## # A tibble: 4,521 x 19
##      age job           marital default balance housing loan  contact  
##    <int> <fct>         <fct>   <fct>     <dbl> <fct>   <fct> <fct>    
##  1    34 technician    married no          297 yes     no    cellular 
##  2    49 services      married no          180 yes     yes   unknown  
##  3    38 admin.        single  no          262 no      no    cellular 
##  4    47 services      married no          367 yes     no    cellular 
##  5    51 self-employed single  no         1640 yes     no    unknown  
##  6    40 unemployed    married no         3382 yes     no    unknown  
##  7    58 retired       married no         1227 no      no    cellular 
##  8    32 unemployed    married no          309 yes     no    telephone
##  9    46 blue-collar   married no          922 yes     no    telephone
## 10    32 services      married no            0 no      no    cellular 
##      day month duration campaign pdays previous poutcome y     education
##    <int> <fct>    <int>    <int> <int>    <int> <fct>    <fct> <fct>    
##  1    29 jan        375        2    -1        0 unknown  0     upper    
##  2     2 jun        392        3    -1        0 unknown  0     upper    
##  3     3 feb        315        2   180        6 failure  1     upper    
##  4    12 may        309        1   306        4 success  1     upper    
##  5    15 may         67        4    -1        0 unknown  0     upper    
##  6    14 may        125        1    -1        0 unknown  0     upper    
##  7    14 aug        182        2    37        2 failure  0     upper    
##  8    13 may        185        1   370        3 failure  0     primary  
##  9    18 nov        296        2    -1        0 unknown  0     upper    
## 10    21 nov         80        1    -1        0 unknown  0     upper    
##    education_unknown education_upper
##                <dbl>           <dbl>
##  1                 0               1
##  2                 0               1
##  3                 0               1
##  4                 0               1
##  5                 0               1
##  6                 0               1
##  7                 0               1
##  8                 0               0
##  9                 0               1
## 10                 0               1
## # ... with 4,511 more rows

Quantile Binning

Quantile binning aims to bin the data into roughly equal groups using quantiles.

bins <- rbin_quantiles(mbank, y, age, 10)
bins

## Binning Summary
## -----------------------------
## Method               Quantile 
## Response             y 
## Predictor            age 
## Bins                 10 
## Count                4521 
## Goods                517 
## Bads                 4004 
## Entropy              0.5 
## Information Value    0.12 
## 
## 
## # A tibble: 10 x 7
##    cut_point bin_count  good   bad      woe         iv entropy
##    <chr>         <int> <int> <int>    <dbl>      <dbl>   <dbl>
##  1 < 29            410    71   339 -0.484   0.0255       0.665
##  2 < 31            313    41   272 -0.155   0.00176      0.560
##  3 < 34            567    55   512  0.184   0.00395      0.459
##  4 < 36            396    45   351  0.00712 0.00000443   0.511
##  5 < 39            519    47   472  0.260   0.00701      0.438
##  6 < 42            431    33   398  0.443   0.0158       0.390
##  7 < 46            449    47   402  0.0993  0.000942     0.484
##  8 < 51            521    40   481  0.440   0.0188       0.391
##  9 < 56            445    49   396  0.0426  0.000176     0.500
## 10 >= 56           470    89   381 -0.593   0.0456       0.700

Plot

plot(bins)

Winsorized Binning

Winsorized binning is similar to equal length binning except that both tails are cut off to obtain a smooth binning result. This technique is often used to remove outliers during the data pre-processing stage. For Winsorized binning, the Winsorized statistics are computed first. After the minimum and maximum have been found, the split points are calculated the same way as in equal length binning.

bins <- rbin_winsorize(mbank, y, age, 10, winsor_rate = 0.05)
bins

## Binning Summary
## ------------------------------
## Method               Winsorize 
## Response             y 
## Predictor            age 
## Bins                 10 
## Count                4521 
## Goods                517 
## Bads                 4004 
## Entropy              0.51 
## Information Value    0.1 
## 
## 
## # A tibble: 10 x 7
##    cut_point bin_count  good   bad    woe       iv entropy
##    <chr>         <int> <int> <int>  <dbl>    <dbl>   <dbl>
##  1 < 30.2          723   112   611 -0.350 0.0224     0.622
##  2 < 33.4          567    55   512  0.184 0.00395    0.459
##  3 < 36.6          573    58   515  0.137 0.00225    0.473
##  4 < 39.8          497    44   453  0.285 0.00798    0.432
##  5 < 43            396    37   359  0.225 0.00408    0.448
##  6 < 46.2          461    43   418  0.227 0.00482    0.447
##  7 < 49.4          281    22   259  0.419 0.00927    0.396
##  8 < 52.6          309    32   277  0.111 0.000811   0.480
##  9 < 55.8          244    25   219  0.123 0.000781   0.477
## 10 >= 55.8         470    89   381 -0.593 0.0456     0.700

Plot

plot(bins)

Learning More

The rbin website includes comprehensive documentation on using the package, including the following article that gives a brief introduction to rbin:

Introduction to rbin

Feedback

rbin has been on CRAN for a few months now while we were fixing bugs and making the API stable. All feedback is welcome. Issues (bugs and feature requests) can be posted to github tracker. For help with code or other related questions, feel free to reach out to us at pkgs@rsquaredacademy.com.

Introducing blorr

Tue, 26 Feb 2019 00:00:00 +0000

I am pleased to announce the blorr package, a set of tools for building and validating binary logistic regression models in R, designed keeping in mind beginner/intermediate R users. The package includes:

comprehensive regression output
variable selection procedures
bivariate analysis, model fit statistics and model validation tools
various plots and underlying data

If you know how to build models using glm(), you will find blorr very useful. Most of the functions use an object of class glm as input. So you just need to build a model using glm() and then pass it onto the functions in blorr. Once you have picked up enough knowledge of R, you can move on to more intuitive approach offered by tidymodels etc. as they offer more flexibility, which blorr does not.

Installation

# Install release version from CRAN
install.packages("blorr")

# Install development version from GitHub
# install.packages("devtools")
devtools::install_github("rsquaredacademy/blorr")

Shiny App

blorr includes a shiny app which can be launched using

blr_launch_app()

or try the live version here.

Read on to learn more about the features of blorr, or see the blorr website for detailed documentation on using the package.

Data

To demonstrate the features of blorr, we will use the bank marketing data set. The data is related with direct marketing campaigns of a Portuguese banking institution. The marketing campaigns were based on phone calls. Often, more than one contact to the same client was required, in order to access if the product (bank term deposit) would be (‘yes’) or not (‘no’) subscribed. It contains a random sample (~4k) of the original data set which can be found at https://archive.ics.uci.edu/ml/datasets/bank+marketing.

Bivariate Analysis

Let us begin with careful bivariate analysis of each possible variable and the outcome variable. We will use information value and likelihood ratio chi square test for selecting the initial set of predictors for our model. The bivariate analysis is currently avialable for categorical predictors only.

blr_bivariate_analysis(bank_marketing, y, job, marital, education, default, 
  housing, loan, contact, poutcome)

##                           Bivariate Analysis                           
## ----------------------------------------------------------------------
## Variable     Information Value    LR Chi Square    LR DF    LR p-value 
## ----------------------------------------------------------------------
##    job             0.16              75.2690        11        0.0000   
##  marital           0.05              21.6821         2        0.0000   
## education          0.05              25.0466         3        0.0000   
##  default           0.02              6.0405          1        0.0140   
##  housing           0.16              72.2813         1        0.0000   
##   loan             0.06              26.6615         1        0.0000   
##  contact           0.31             124.3834         2        0.0000   
## poutcome           0.53             270.6450         3        0.0000   
## ----------------------------------------------------------------------

Weight of Evidence & Information Value

Weight of evidence (WoE) is used to assess the relative risk of di¤erent attributes for a characteristic and as a means to transform characteristics into variables. It is also a very useful tool for binning. The WoE for any group with average odds is zero. A negative WoE indicates that the proportion of defaults is higher for that attribute than the overall proportion and indicates higher risk.

The information value is used to rank order variables in terms of their predictive power. A high information value indicates a high ability to discriminate. Values for the information value will always be positive and may be above 3 when assessing highly predictive characteristics. Characteristics with information values less than 0:10 are typically viewed as weak, while values over 0.30 are sought after.

blr_woe_iv(bank_marketing, job, y)

##                                Weight of Evidence                                
## --------------------------------------------------------------------------------
##    levels        0s_count    1s_count    0s_dist    1s_dist        woe      iv   
## --------------------------------------------------------------------------------
##  management        809         130          0.20       0.25      -0.22     0.01  
##  technician        682          79          0.17       0.15       0.11     0.00  
## entrepreneur       139          12          0.03       0.02       0.40     0.00  
##  blue-collar       937          73          0.23       0.14       0.51     0.05  
##    unknown          29          2           0.01       0.00       0.61     0.00  
##    retired         152          47          0.04       0.09      -0.87     0.05  
##    admin.          433          61          0.11       0.12      -0.09     0.00  
##   services         392          39          0.10       0.08       0.26     0.01  
## self-employed      132          22          0.03       0.04      -0.26     0.00  
##  unemployed        126          15          0.03       0.03       0.08     0.00  
##   housemaid        110          12          0.03       0.02       0.17     0.00  
##    student          63          25          0.02       0.05      -1.13     0.04  
## --------------------------------------------------------------------------------
## 
##       Information Value       
## -----------------------------
## Variable    Information Value 
## -----------------------------
##   job            0.1594       
## -----------------------------

Plot

k <- blr_woe_iv(bank_marketing, job, y)
plot(k)

Multiple Variables

We can generate the weight of evidence and information value for multiple variables using blr_woe_iv_stats().

blr_woe_iv_stats(bank_marketing, y, job, marital, education)

## Variable: job
## 
##                                Weight of Evidence                                
## --------------------------------------------------------------------------------
##    levels        0s_count    1s_count    0s_dist    1s_dist        woe      iv   
## --------------------------------------------------------------------------------
##  management        809         130          0.20       0.25      -0.22     0.01  
##  technician        682          79          0.17       0.15       0.11     0.00  
## entrepreneur       139          12          0.03       0.02       0.40     0.00  
##  blue-collar       937          73          0.23       0.14       0.51     0.05  
##    unknown          29          2           0.01       0.00       0.61     0.00  
##    retired         152          47          0.04       0.09      -0.87     0.05  
##    admin.          433          61          0.11       0.12      -0.09     0.00  
##   services         392          39          0.10       0.08       0.26     0.01  
## self-employed      132          22          0.03       0.04      -0.26     0.00  
##  unemployed        126          15          0.03       0.03       0.08     0.00  
##   housemaid        110          12          0.03       0.02       0.17     0.00  
##    student          63          25          0.02       0.05      -1.13     0.04  
## --------------------------------------------------------------------------------
## 
##       Information Value       
## -----------------------------
## Variable    Information Value 
## -----------------------------
##   job            0.1594       
## -----------------------------
## 
## 
## Variable: marital
## 
##                             Weight of Evidence                              
## ---------------------------------------------------------------------------
##  levels     0s_count    1s_count    0s_dist    1s_dist        woe      iv   
## ---------------------------------------------------------------------------
## married       2467        273          0.62       0.53       0.15     0.01  
##  single       1079        191          0.27       0.37      -0.32     0.03  
## divorced      458          53          0.11       0.10       0.11     0.00  
## ---------------------------------------------------------------------------
## 
##       Information Value       
## -----------------------------
## Variable    Information Value 
## -----------------------------
## marital          0.0464       
## -----------------------------
## 
## 
## Variable: education
## 
##                              Weight of Evidence                              
## ----------------------------------------------------------------------------
##  levels      0s_count    1s_count    0s_dist    1s_dist        woe      iv   
## ----------------------------------------------------------------------------
## tertiary       1104        195          0.28       0.38      -0.31     0.03  
## secondary      2121        231          0.53       0.45       0.17     0.01  
##  unknown       154          25          0.04       0.05      -0.23     0.00  
##  primary       625          66          0.16       0.13       0.20     0.01  
## ----------------------------------------------------------------------------
## 
##       Information Value        
## ------------------------------
## Variable     Information Value 
## ------------------------------
## education         0.0539       
## ------------------------------

blr_woe_iv() and blr_woe_iv_stats() are currently avialable for categorical predictors only.

Stepwise Selection

For the initial/ first cut model, all the independent variables are put into the model. Our goal is to include a limited number of independent variables (5-15) which are all significant, without sacrificing too much on the model performance. The rationale behind not-including too many variables is that the model would be over fitted and would become unstable when tested on the validation sample. The variable reduction is done using forward or backward or stepwise variable selection procedures. We will use blr_step_aic_both() to shortlist predictors for our model.

Model

model <- glm(y ~ ., data = bank_marketing, family = binomial(link = 'logit'))

Selection Summary

blr_step_aic_both(model)

## Stepwise Selection Method 
## -------------------------
## 
## Candidate Terms: 
## 
## 1 . age 
## 2 . job 
## 3 . marital 
## 4 . education 
## 5 . default 
## 6 . balance 
## 7 . housing 
## 8 . loan 
## 9 . contact 
## 10 . day 
## 11 . month 
## 12 . duration 
## 13 . campaign 
## 14 . pdays 
## 15 . previous 
## 16 . poutcome 
## 
## 
## Variables Entered/Removed: 
## 
## - duration added 
## - poutcome added 
## - month added 
## - contact added 
## - housing added 
## - loan added 
## - campaign added 
## - marital added 
## - education added 
## - age added 
## 
## No more variables to be added or removed.

## 
## 
##                      Stepwise Summary                      
## ---------------------------------------------------------
## Variable      Method       AIC         BIC       Deviance 
## ---------------------------------------------------------
## duration     addition    2674.384    2687.217    2670.384 
## poutcome     addition    2396.014    2428.097    2386.014 
## month        addition    2274.109    2376.773    2242.109 
## contact      addition    2207.884    2323.381    2171.884 
## housing      addition    2184.550    2306.463    2146.550 
## loan         addition    2171.972    2300.302    2131.972 
## campaign     addition    2164.164    2298.910    2122.164 
## marital      addition    2158.524    2306.103    2112.524 
## education    addition    2155.837    2322.666    2103.837 
## age          addition    2154.272    2327.517    2100.272 
## ---------------------------------------------------------

Plot

model %>%
  blr_step_aic_both() %>%
  plot()

## Stepwise Selection Method 
## -------------------------
## 
## Candidate Terms: 
## 
## 1 . age 
## 2 . job 
## 3 . marital 
## 4 . education 
## 5 . default 
## 6 . balance 
## 7 . housing 
## 8 . loan 
## 9 . contact 
## 10 . day 
## 11 . month 
## 12 . duration 
## 13 . campaign 
## 14 . pdays 
## 15 . previous 
## 16 . poutcome 
## 
## 
## Variables Entered/Removed: 
## 
## - duration added 
## - poutcome added 
## - month added 
## - contact added 
## - housing added 
## - loan added 
## - campaign added 
## - marital added 
## - education added 
## - age added 
## 
## No more variables to be added or removed.

Regression Output

Model

We can use bivariate analysis and stepwise selection procedure to shortlist predictors and build the model using the glm(). The predictors used in the below model are for illustration purposes and not necessarily shortlisted from the bivariate analysis and variable selection procedures.

model <- glm(y ~  age + duration + previous + housing + default +
             loan + poutcome + job + marital, data = bank_marketing, 
             family = binomial(link = 'logit'))

Use blr_regress() to generate comprehensive regression output. It accepts either of the following

model built using glm()
model formula and data

Using Model

Let us look at the output generated from blr_regress():

blr_regress(model)

## - Creating model overview. 
## - Creating response profile. 
## - Extracting maximum likelihood estimates. 
## - Estimating concordant and discordant pairs.

##                              Model Overview                              
## ------------------------------------------------------------------------
## Data Set    Resp Var    Obs.    Df. Model    Df. Residual    Convergence 
## ------------------------------------------------------------------------
##   data         y        4521      4520           4498           TRUE     
## ------------------------------------------------------------------------
## 
##                     Response Summary                     
## --------------------------------------------------------
## Outcome        Frequency        Outcome        Frequency 
## --------------------------------------------------------
##    0             4004              1              517    
## --------------------------------------------------------
## 
##                      Maximum Likelihood Estimates                       
## -----------------------------------------------------------------------
##    Parameter        DF    Estimate    Std. Error    z value     Pr(>|z|) 
## -----------------------------------------------------------------------
##   (Intercept)       1     -5.1347        0.3728    -13.7729      0.0000 
##       age           1      0.0096        0.0067      1.4299      0.1528 
##     duration        1      0.0042         2e-04     20.7853      0.0000 
##     previous        1     -0.0357        0.0392     -0.9089      0.3634 
##    housingno        1      0.7894        0.1232      6.4098      0.0000 
##    defaultyes       1     -0.8691        0.6919     -1.2562      0.2091 
##      loanno         1      0.6598        0.1945      3.3925       7e-04 
## poutcomefailure     1      0.6085        0.2012      3.0248      0.0025 
##  poutcomeother      1      1.1354        0.2700      4.2057      0.0000 
## poutcomesuccess     1      3.2481        0.2462     13.1913      0.0000 
##  jobtechnician      1     -0.2713        0.1806     -1.5019      0.1331 
## jobentrepreneur     1     -0.7041        0.3809     -1.8486      0.0645 
##  jobblue-collar     1     -0.6132        0.1867     -3.2851      0.0010 
##    jobunknown       1     -0.9932        0.8226     -1.2073      0.2273 
##    jobretired       1      0.3197        0.2729      1.1713      0.2415 
##    jobadmin.        1      0.1120        0.2001      0.5599      0.5755 
##   jobservices       1     -0.1750        0.2265     -0.7728      0.4397 
## jobself-employed    1     -0.1408        0.3009     -0.4680      0.6398 
##  jobunemployed      1     -0.6581        0.3432     -1.9174      0.0552 
##   jobhousemaid      1     -0.7456        0.3932     -1.8963      0.0579 
##    jobstudent       1      0.1927        0.3433      0.5613      0.5746 
##  maritalsingle      1      0.5451        0.1387      3.9299       1e-04 
## maritaldivorced     1     -0.1989        0.1986     -1.0012      0.3167 
## -----------------------------------------------------------------------
## 
##  Association of Predicted Probabilities and Observed Responses  
## ---------------------------------------------------------------
## % Concordant          0.8886          Somers' D        0.7773   
## % Discordant          0.1114          Gamma            0.7773   
## % Tied                0.0000          Tau-a            0.1575   
## Pairs                2070068          c                0.8886   
## ---------------------------------------------------------------

If you want to examine the odds ratio estimates, set odd_conf_limit to TRUE. The odds ratio estimates are not explicitly computed as we observed considerable increase in computation time when dealing with large data sets.

Using Formula

Let us use the model formula and the data set to generate the above results.

blr_regress(y ~  age + duration + previous + housing + default +
             loan + poutcome + job + marital, data = bank_marketing)

## - Creating model overview. 
## - Creating response profile. 
## - Extracting maximum likelihood estimates. 
## - Estimating concordant and discordant pairs.

##                              Model Overview                              
## ------------------------------------------------------------------------
## Data Set    Resp Var    Obs.    Df. Model    Df. Residual    Convergence 
## ------------------------------------------------------------------------
##   data         y        4521      4520           4498           TRUE     
## ------------------------------------------------------------------------
## 
##                     Response Summary                     
## --------------------------------------------------------
## Outcome        Frequency        Outcome        Frequency 
## --------------------------------------------------------
##    0             4004              1              517    
## --------------------------------------------------------
## 
##                      Maximum Likelihood Estimates                       
## -----------------------------------------------------------------------
##    Parameter        DF    Estimate    Std. Error    z value     Pr(>|z|) 
## -----------------------------------------------------------------------
##   (Intercept)       1     -5.1347        0.3728    -13.7729      0.0000 
##       age           1      0.0096        0.0067      1.4299      0.1528 
##     duration        1      0.0042         2e-04     20.7853      0.0000 
##     previous        1     -0.0357        0.0392     -0.9089      0.3634 
##    housingno        1      0.7894        0.1232      6.4098      0.0000 
##    defaultyes       1     -0.8691        0.6919     -1.2562      0.2091 
##      loanno         1      0.6598        0.1945      3.3925       7e-04 
## poutcomefailure     1      0.6085        0.2012      3.0248      0.0025 
##  poutcomeother      1      1.1354        0.2700      4.2057      0.0000 
## poutcomesuccess     1      3.2481        0.2462     13.1913      0.0000 
##  jobtechnician      1     -0.2713        0.1806     -1.5019      0.1331 
## jobentrepreneur     1     -0.7041        0.3809     -1.8486      0.0645 
##  jobblue-collar     1     -0.6132        0.1867     -3.2851      0.0010 
##    jobunknown       1     -0.9932        0.8226     -1.2073      0.2273 
##    jobretired       1      0.3197        0.2729      1.1713      0.2415 
##    jobadmin.        1      0.1120        0.2001      0.5599      0.5755 
##   jobservices       1     -0.1750        0.2265     -0.7728      0.4397 
## jobself-employed    1     -0.1408        0.3009     -0.4680      0.6398 
##  jobunemployed      1     -0.6581        0.3432     -1.9174      0.0552 
##   jobhousemaid      1     -0.7456        0.3932     -1.8963      0.0579 
##    jobstudent       1      0.1927        0.3433      0.5613      0.5746 
##  maritalsingle      1      0.5451        0.1387      3.9299       1e-04 
## maritaldivorced     1     -0.1989        0.1986     -1.0012      0.3167 
## -----------------------------------------------------------------------
## 
##  Association of Predicted Probabilities and Observed Responses  
## ---------------------------------------------------------------
## % Concordant          0.8886          Somers' D        0.7773   
## % Discordant          0.1114          Gamma            0.7773   
## % Tied                0.0000          Tau-a            0.1575   
## Pairs                2070068          c                0.8886   
## ---------------------------------------------------------------

Model Fit Statistics

Model fit statistics are available to assess how well the model fits the data and to compare two different models.The output includes likelihood ratio test, AIC, BIC and a host of pseudo r-squared measures. You can read more about pseudo r-squared at https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-what-are-pseudo-r-squareds/.

Single Model

blr_model_fit_stats(model)

##                                Model Fit Statistics                                
## ----------------------------------------------------------------------------------
## Log-Lik Intercept Only:     -1607.330    Log-Lik Full Model:             -1123.340 
## Deviance(4498):              2246.679    LR(22):                           967.980 
##                                          Prob > LR:                          0.000 
## MCFadden's R2                   0.301    McFadden's Adj R2:                  0.287 
## ML (Cox-Snell) R2:              0.193    Cragg-Uhler(Nagelkerke) R2:         0.379 
## McKelvey & Zavoina's R2:        0.388    Efron's R2:                         0.278 
## Count R2:                       0.904    Adj Count R2:                       0.157 
## BIC:                         2440.259    AIC:                             2292.679 
## ----------------------------------------------------------------------------------

Model Validation

Hosmer Lemeshow Test

Hosmer and Lemeshow developed a goodness-of-fit test for logistic regression models with binary responses. The test involves dividing the data into approximately ten groups of roughly equal size based on the percentiles of the estimated probabilities. The observations are sorted in increasing order of their estimated probability of having an even outcome. The discrepancies between the observed and expected number of observations in these groups are summarized by the Pearson chi-square statistic, which is then compared to chi-square distribution with t degrees of freedom, where t is the number of groups minus 2. Lower values of Goodness-of-fit are preferred.

blr_test_hosmer_lemeshow(model)

##            Partition for the Hosmer & Lemeshow Test            
## --------------------------------------------------------------
##                         def = 1                 def = 0        
## Group    Total    Observed    Expected    Observed    Expected 
## --------------------------------------------------------------
##   1       453        2          5.14        451        447.86  
##   2       452        3          8.63        449        443.37  
##   3       452        4         11.88        448        440.12  
##   4       452        7         15.29        445        436.71  
##   5       452        14        19.39        438        432.61  
##   6       452        10        24.97        442        427.03  
##   7       452        31        33.65        421        418.35  
##   8       452        62        49.74        390        402.26  
##   9       452       128        88.10        324        363.90  
##  10       452       256        260.21       196        191.79  
## --------------------------------------------------------------
## 
##      Goodness of Fit Test      
## ------------------------------
## Chi-Square    DF    Pr > ChiSq 
## ------------------------------
##  52.9942      8       0.0000   
## ------------------------------

Gains Table & Lift Chart

A lift curve is a graphical representation of the % of cumulative events captured at a specific cut-off. The cut-off can be a particular decile or a percentile. Similar, to rank ordering procedure, the data is in descending order of the scores and is then grouped into deciles/percentiles. The cumulative number of observations and events are then computed for each decile/percentile. The lift curve is the created using the cumulative % population as the x-axis and the cumulative percentage of events as the y-axis.

blr_gains_table(model)

## # A tibble: 10 x 12
##    decile total   `1`   `0`    ks    tp    tn    fp    fn sensitivity
##     <dbl> <int> <int> <int> <dbl> <int> <int> <int> <int>       <dbl>
##  1      1   452   256   196  44.6   256  3808   196   261        49.5
##  2      2   452   128   324  61.3   384  3484   520   133        74.3
##  3      3   452    62   390  63.5   446  3094   910    71        86.3
##  4      4   452    31   421  59.0   477  2673  1331    40        92.3
##  5      5   452    10   442  49.9   487  2231  1773    30        94.2
##  6      6   452    14   438  41.7   501  1793  2211    16        96.9
##  7      7   452     7   445  31.9   508  1348  2656     9        98.3
##  8      8   452     4   448  21.5   512   900  3104     5        99.0
##  9      9   452     3   449  10.9   515   451  3553     2        99.6
## 10     10   453     2   451   0     517     0  4004     0       100  
## # ... with 2 more variables: specificity <dbl>, accuracy <dbl>

Lift Chart

model %>%
    blr_gains_table() %>%
    plot()

ROC Curve

ROC curve is a graphical representation of the validity of cut-offs for a logistic regression model. The ROC curve is plotted using the sensitivity and specificity for all possible cut-offs, i.e., all the probability scores. The graph is plotted using sensitivity on the y-axis and 1-specificity on the x-axis. Any point on the ROC curve represents a sensitivity X (1-specificity) measure corresponding to a cut-off. The area under the ROC curve is used as a validation measure for the model – the bigger the area the better is the model.

model %>%
    blr_gains_table() %>%
  blr_roc_curve()

KS Chart

The KS Statistic is again a measure of model efficiency, and it is created using the lift curve. The lift curve is created to plot % events. If we also plot % non-events on the same scale, with % population at x-axis, we would get another curve. The maximum distance between the lift curve for events and that for non-events is termed as KS. For a good model, KS should be big (>=0.3) and should occur as close to the event rate as possible.

model %>%
    blr_gains_table() %>%
  blr_ks_chart()

Decile Lift Chart

The decile lift chart displays the lift over the global mean event rate for each decile. For a model with good discriminatory power, the top deciles should have an event/conversion rate greater than the global mean.

model %>%
  blr_gains_table() %>%
  blr_decile_lift_chart()

Capture Rate by Decile

If the model has good discriminatory power, the top deciles should have a higher event/conversion rate compared to the bottom deciles.

model %>%
  blr_gains_table() %>%
  blr_decile_capture_rate()

Lorenz Curve

The Lorenz curve is a simple graphic device which illustrates the degree of inequality in the distribution of thevariable concerned. It is a visual representation of inequality used to measure the discriminatory power of the predictive model.

blr_lorenz_curve(model)

Residual & Influence Diagnostics

blorr can generate 22 plots for residual, influence and leverage diagnostics.

Influence Diagnostics

blr_plot_diag_influence(model)

Leverage Diagnostics

blr_plot_diag_leverage(model)

Fitted Values Diagnostics

blr_plot_diag_fit(model)

Learning More

The blorr website includes comprehensive documentation on using the package, including the following article that covers various aspects of using blorr.

Feedback

All feedback is welcome. Issues (bugs and feature requests) can be posted to github tracker. For help with code or other related questions, feel free to reach me hebbali.aravind@gmail.com.

Descriptive Statistics in R

Tue, 19 Feb 2019 00:00:00 +0000

I am pleased to announce the descriptr package, a set of tools for generating descriptive/summary statistics.

Installation

# Install release version from CRAN
install.packages("descriptr")

# Install development version from GitHub
# install.packages("devtools")
devtools::install_github("rsquaredacademy/descriptr")

Shiny App

descriptr includes a shiny app which can be launched using

ds_launch_shiny_app()

or try the live version here.

Read on to learn more about the features of descriptr, or see the descriptr website for detailed documentation on using the package.

Data

We have modified the mtcars data to create a new data set mtcarz. The only difference between the two data sets is related to the variable types.

str(mtcarz)

## 'data.frame':    32 obs. of  11 variables:
##  $ mpg : num  21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ...
##  $ cyl : Factor w/ 3 levels "4","6","8": 2 2 1 2 3 2 3 1 1 2 ...
##  $ disp: num  160 160 108 258 360 ...
##  $ hp  : num  110 110 93 110 175 105 245 62 95 123 ...
##  $ drat: num  3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ...
##  $ wt  : num  2.62 2.88 2.32 3.21 3.44 ...
##  $ qsec: num  16.5 17 18.6 19.4 17 ...
##  $ vs  : Factor w/ 2 levels "0","1": 1 1 2 2 1 2 1 2 2 2 ...
##  $ am  : Factor w/ 2 levels "0","1": 2 2 2 1 1 1 1 1 1 1 ...
##  $ gear: Factor w/ 3 levels "3","4","5": 2 2 2 1 1 1 1 2 2 2 ...
##  $ carb: Factor w/ 6 levels "1","2","3","4",..: 4 4 1 1 2 1 4 2 2 4 ...

Data Screening

The ds_screener() function will screen a data set and return the following: - Column/Variable Names - Data Type - Levels (in case of categorical data) - Number of missing observations - % of missing observations

ds_screener(mtcarz)

## -----------------------------------------------------------------------
## |  Column Name  |  Data Type  |  Levels   |  Missing  |  Missing (%)  |
## -----------------------------------------------------------------------
## |      mpg      |   numeric   |    NA     |     0     |       0       |
## |      cyl      |   factor    |   4 6 8   |     0     |       0       |
## |     disp      |   numeric   |    NA     |     0     |       0       |
## |      hp       |   numeric   |    NA     |     0     |       0       |
## |     drat      |   numeric   |    NA     |     0     |       0       |
## |      wt       |   numeric   |    NA     |     0     |       0       |
## |     qsec      |   numeric   |    NA     |     0     |       0       |
## |      vs       |   factor    |    0 1    |     0     |       0       |
## |      am       |   factor    |    0 1    |     0     |       0       |
## |     gear      |   factor    |   3 4 5   |     0     |       0       |
## |     carb      |   factor    |1 2 3 4 6 8|     0     |       0       |
## -----------------------------------------------------------------------
## 
##  Overall Missing Values           0 
##  Percentage of Missing Values     0 %
##  Rows with Missing Values         0 
##  Columns With Missing Values      0

Continuous Data

Summary Statistics

The ds_summary_stats function returns a comprehensive set of statistics including measures of location, variation, symmetry and extreme observations.

ds_summary_stats(mtcarz, mpg)

## ------------------------------ Variable: mpg ------------------------------
## 
##                         Univariate Analysis                          
## 
##  N                       32.00      Variance                36.32 
##  Missing                  0.00      Std Deviation            6.03 
##  Mean                    20.09      Range                   23.50 
##  Median                  19.20      Interquartile Range      7.38 
##  Mode                    10.40      Uncorrected SS       14042.31 
##  Trimmed Mean            19.95      Corrected SS          1126.05 
##  Skewness                 0.67      Coeff Variation         30.00 
##  Kurtosis                -0.02      Std Error Mean           1.07 
## 
##                               Quantiles                               
## 
##               Quantile                            Value                
## 
##              Max                                  33.90                
##              99%                                  33.44                
##              95%                                  31.30                
##              90%                                  30.09                
##              Q3                                   22.80                
##              Median                               19.20                
##              Q1                                   15.43                
##              10%                                  14.34                
##              5%                                   12.00                
##              1%                                   10.40                
##              Min                                  10.40                
## 
##                             Extreme Values                            
## 
##                 Low                                High                
## 
##   Obs                        Value       Obs                        Value 
##   15                         10.4        20                         33.9  
##   16                         10.4        18                         32.4  
##   24                         13.3        19                         30.4  
##    7                         14.3        28                         30.4  
##   17                         14.7        26                         27.3

You can pass multiple variables as shown below:

ds_summary_stats(mtcarz, mpg, disp)

## ------------------------------ Variable: mpg ------------------------------
## 
##                         Univariate Analysis                          
## 
##  N                       32.00      Variance                36.32 
##  Missing                  0.00      Std Deviation            6.03 
##  Mean                    20.09      Range                   23.50 
##  Median                  19.20      Interquartile Range      7.38 
##  Mode                    10.40      Uncorrected SS       14042.31 
##  Trimmed Mean            19.95      Corrected SS          1126.05 
##  Skewness                 0.67      Coeff Variation         30.00 
##  Kurtosis                -0.02      Std Error Mean           1.07 
## 
##                               Quantiles                               
## 
##               Quantile                            Value                
## 
##              Max                                  33.90                
##              99%                                  33.44                
##              95%                                  31.30                
##              90%                                  30.09                
##              Q3                                   22.80                
##              Median                               19.20                
##              Q1                                   15.43                
##              10%                                  14.34                
##              5%                                   12.00                
##              1%                                   10.40                
##              Min                                  10.40                
## 
##                             Extreme Values                            
## 
##                 Low                                High                
## 
##   Obs                        Value       Obs                        Value 
##   15                         10.4        20                         33.9  
##   16                         10.4        18                         32.4  
##   24                         13.3        19                         30.4  
##    7                         14.3        28                         30.4  
##   17                         14.7        26                         27.3  
## 
## 
## 
## ------------------------------ Variable: disp -----------------------------
## 
##                           Univariate Analysis                            
## 
##  N                         32.00      Variance               15360.80 
##  Missing                    0.00      Std Deviation            123.94 
##  Mean                     230.72      Range                    400.90 
##  Median                   196.30      Interquartile Range      205.18 
##  Mode                     275.80      Uncorrected SS       2179627.47 
##  Trimmed Mean             228.00      Corrected SS          476184.79 
##  Skewness                   0.42      Coeff Variation           53.72 
##  Kurtosis                  -1.07      Std Error Mean            21.91 
## 
##                                 Quantiles                                 
## 
##                Quantile                              Value                 
## 
##               Max                                    472.00                
##               99%                                    468.28                
##               95%                                    449.00                
##               90%                                    396.00                
##               Q3                                     326.00                
##               Median                                 196.30                
##               Q1                                     120.83                
##               10%                                    80.61                 
##               5%                                     77.35                 
##               1%                                     72.53                 
##               Min                                    71.10                 
## 
##                               Extreme Values                              
## 
##                  Low                                  High                 
## 
##   Obs                          Value       Obs                          Value 
##   20                           71.1        15                            472  
##   19                           75.7        16                            460  
##   18                           78.7        17                            440  
##   26                            79         25                            400  
##   28                           95.1         5                            360

If you do not specify any variables, it will detect all the continuous variables in the data set and return summary statistics for each of them.

Frequency Distribution

The ds_freq_table function creates frequency tables for continuous variables. The default number of intervals is 5.

ds_freq_table(mtcarz, mpg, 4)

##                                 Variable: mpg                                 
## |---------------------------------------------------------------------------|
## |      Bins       | Frequency | Cum Frequency |   Percent    | Cum Percent  |
## |---------------------------------------------------------------------------|
## |  10.4  -  16.3  |    10     |      10       |    31.25     |    31.25     |
## |---------------------------------------------------------------------------|
## |  16.3  -  22.1  |    13     |      23       |    40.62     |    71.88     |
## |---------------------------------------------------------------------------|
## |  22.1  -   28   |     5     |      28       |    15.62     |     87.5     |
## |---------------------------------------------------------------------------|
## |   28   -  33.9  |     4     |      32       |     12.5     |     100      |
## |---------------------------------------------------------------------------|
## |      Total      |    32     |       -       |    100.00    |      -       |
## |---------------------------------------------------------------------------|

Histogram

A plot() method has been defined which will generate a histogram.

k <- ds_freq_table(mtcarz, mpg, 4)
plot(k)

Auto Summary

If you want to view summary statistics and frequency tables of all or subset of variables in a data set, use ds_auto_summary().

ds_auto_summary_stats(mtcarz, disp, mpg)

## ------------------------------ Variable: disp -----------------------------
## 
## ---------------------------- Summary Statistics ---------------------------
## 
## ------------------------------ Variable: disp -----------------------------
## 
##                           Univariate Analysis                            
## 
##  N                         32.00      Variance               15360.80 
##  Missing                    0.00      Std Deviation            123.94 
##  Mean                     230.72      Range                    400.90 
##  Median                   196.30      Interquartile Range      205.18 
##  Mode                     275.80      Uncorrected SS       2179627.47 
##  Trimmed Mean             228.00      Corrected SS          476184.79 
##  Skewness                   0.42      Coeff Variation           53.72 
##  Kurtosis                  -1.07      Std Error Mean            21.91 
## 
##                                 Quantiles                                 
## 
##                Quantile                              Value                 
## 
##               Max                                    472.00                
##               99%                                    468.28                
##               95%                                    449.00                
##               90%                                    396.00                
##               Q3                                     326.00                
##               Median                                 196.30                
##               Q1                                     120.83                
##               10%                                    80.61                 
##               5%                                     77.35                 
##               1%                                     72.53                 
##               Min                                    71.10                 
## 
##                               Extreme Values                              
## 
##                  Low                                  High                 
## 
##   Obs                          Value       Obs                          Value 
##   20                           71.1        15                            472  
##   19                           75.7        16                            460  
##   18                           78.7        17                            440  
##   26                            79         25                            400  
##   28                           95.1         5                            360  
## 
## 
## 
## NULL
## 
## 
## -------------------------- Frequency Distribution -------------------------
## 
##                                Variable: disp                                 
## |---------------------------------------------------------------------------|
## |      Bins       | Frequency | Cum Frequency |   Percent    | Cum Percent  |
## |---------------------------------------------------------------------------|
## |  71.1  - 151.3  |    12     |      12       |     37.5     |     37.5     |
## |---------------------------------------------------------------------------|
## | 151.3  - 231.5  |     5     |      17       |    15.62     |    53.12     |
## |---------------------------------------------------------------------------|
## | 231.5  - 311.6  |     6     |      23       |    18.75     |    71.88     |
## |---------------------------------------------------------------------------|
## | 311.6  - 391.8  |     5     |      28       |    15.62     |     87.5     |
## |---------------------------------------------------------------------------|
## | 391.8  -  472   |     4     |      32       |     12.5     |     100      |
## |---------------------------------------------------------------------------|
## |      Total      |    32     |       -       |    100.00    |      -       |
## |---------------------------------------------------------------------------|
## 
## 
## ------------------------------ Variable: mpg ------------------------------
## 
## ---------------------------- Summary Statistics ---------------------------
## 
## ------------------------------ Variable: mpg ------------------------------
## 
##                         Univariate Analysis                          
## 
##  N                       32.00      Variance                36.32 
##  Missing                  0.00      Std Deviation            6.03 
##  Mean                    20.09      Range                   23.50 
##  Median                  19.20      Interquartile Range      7.38 
##  Mode                    10.40      Uncorrected SS       14042.31 
##  Trimmed Mean            19.95      Corrected SS          1126.05 
##  Skewness                 0.67      Coeff Variation         30.00 
##  Kurtosis                -0.02      Std Error Mean           1.07 
## 
##                               Quantiles                               
## 
##               Quantile                            Value                
## 
##              Max                                  33.90                
##              99%                                  33.44                
##              95%                                  31.30                
##              90%                                  30.09                
##              Q3                                   22.80                
##              Median                               19.20                
##              Q1                                   15.43                
##              10%                                  14.34                
##              5%                                   12.00                
##              1%                                   10.40                
##              Min                                  10.40                
## 
##                             Extreme Values                            
## 
##                 Low                                High                
## 
##   Obs                        Value       Obs                        Value 
##   15                         10.4        20                         33.9  
##   16                         10.4        18                         32.4  
##   24                         13.3        19                         30.4  
##    7                         14.3        28                         30.4  
##   17                         14.7        26                         27.3  
## 
## 
## 
## NULL
## 
## 
## -------------------------- Frequency Distribution -------------------------
## 
##                               Variable: mpg                               
## |-----------------------------------------------------------------------|
## |    Bins     | Frequency | Cum Frequency |   Percent    | Cum Percent  |
## |-----------------------------------------------------------------------|
## | 10.4 - 15.1 |     6     |       6       |    18.75     |    18.75     |
## |-----------------------------------------------------------------------|
## | 15.1 - 19.8 |    12     |      18       |     37.5     |    56.25     |
## |-----------------------------------------------------------------------|
## | 19.8 - 24.5 |     8     |      26       |      25      |    81.25     |
## |-----------------------------------------------------------------------|
## | 24.5 - 29.2 |     2     |      28       |     6.25     |     87.5     |
## |-----------------------------------------------------------------------|
## | 29.2 - 33.9 |     4     |      32       |     12.5     |     100      |
## |-----------------------------------------------------------------------|
## |    Total    |    32     |       -       |    100.00    |      -       |
## |-----------------------------------------------------------------------|

Group Summary

The ds_group_summary() function returns descriptive statistics of a continuous variable for the different levels of a categorical variable.

k <- ds_group_summary(mtcarz, cyl, mpg)
k

##                                        mpg by cyl                                         
## -----------------------------------------------------------------------------------------
## |     Statistic/Levels|                    4|                    6|                    8|
## -----------------------------------------------------------------------------------------
## |                  Obs|                   11|                    7|                   14|
## |              Minimum|                 21.4|                 17.8|                 10.4|
## |              Maximum|                 33.9|                 21.4|                 19.2|
## |                 Mean|                26.66|                19.74|                 15.1|
## |               Median|                   26|                 19.7|                 15.2|
## |                 Mode|                 22.8|                   21|                 10.4|
## |       Std. Deviation|                 4.51|                 1.45|                 2.56|
## |             Variance|                20.34|                 2.11|                 6.55|
## |             Skewness|                 0.35|                -0.26|                -0.46|
## |             Kurtosis|                -1.43|                -1.83|                 0.33|
## |       Uncorrected SS|              8023.83|              2741.14|              3277.34|
## |         Corrected SS|               203.39|                12.68|                 85.2|
## |      Coeff Variation|                16.91|                 7.36|                16.95|
## |      Std. Error Mean|                 1.36|                 0.55|                 0.68|
## |                Range|                 12.5|                  3.6|                  8.8|
## |  Interquartile Range|                  7.6|                 2.35|                 1.85|
## -----------------------------------------------------------------------------------------

ds_group_summary() returns a tibble which can be used for further analysis.

k$tidy_stats

## # A tibble: 3 x 15
##   cyl   length   min   max  mean median  mode    sd variance skewness
##   <fct>  <int> <dbl> <dbl> <dbl>  <dbl> <dbl> <dbl>    <dbl>    <dbl>
## 1 4         11  21.4  33.9  26.7   26    22.8  4.51    20.3     0.348
## 2 6          7  17.8  21.4  19.7   19.7  21    1.45     2.11   -0.259
## 3 8         14  10.4  19.2  15.1   15.2  10.4  2.56     6.55   -0.456
## # ... with 5 more variables: kurtosis <dbl>, coeff_var <dbl>,
## #   std_error <dbl>, range <dbl>, iqr <dbl>

Box Plot

A plot() method has been defined for comparing distributions.

k <- ds_group_summary(mtcarz, cyl, mpg)
plot(k)

Multiple Variables

If you want grouped summary statistics for multiple variables in a data set, use ds_auto_group_summary().

ds_auto_group_summary(mtcarz, cyl, gear, mpg)

##                                        mpg by cyl                                         
## -----------------------------------------------------------------------------------------
## |     Statistic/Levels|                    4|                    6|                    8|
## -----------------------------------------------------------------------------------------
## |                  Obs|                   11|                    7|                   14|
## |              Minimum|                 21.4|                 17.8|                 10.4|
## |              Maximum|                 33.9|                 21.4|                 19.2|
## |                 Mean|                26.66|                19.74|                 15.1|
## |               Median|                   26|                 19.7|                 15.2|
## |                 Mode|                 22.8|                   21|                 10.4|
## |       Std. Deviation|                 4.51|                 1.45|                 2.56|
## |             Variance|                20.34|                 2.11|                 6.55|
## |             Skewness|                 0.35|                -0.26|                -0.46|
## |             Kurtosis|                -1.43|                -1.83|                 0.33|
## |       Uncorrected SS|              8023.83|              2741.14|              3277.34|
## |         Corrected SS|               203.39|                12.68|                 85.2|
## |      Coeff Variation|                16.91|                 7.36|                16.95|
## |      Std. Error Mean|                 1.36|                 0.55|                 0.68|
## |                Range|                 12.5|                  3.6|                  8.8|
## |  Interquartile Range|                  7.6|                 2.35|                 1.85|
## -----------------------------------------------------------------------------------------
## 
## 
## 
##                                        mpg by gear                                        
## -----------------------------------------------------------------------------------------
## |     Statistic/Levels|                    3|                    4|                    5|
## -----------------------------------------------------------------------------------------
## |                  Obs|                   15|                   12|                    5|
## |              Minimum|                 10.4|                 17.8|                   15|
## |              Maximum|                 21.5|                 33.9|                 30.4|
## |                 Mean|                16.11|                24.53|                21.38|
## |               Median|                 15.5|                 22.8|                 19.7|
## |                 Mode|                 10.4|                   21|                   15|
## |       Std. Deviation|                 3.37|                 5.28|                 6.66|
## |             Variance|                11.37|                27.84|                44.34|
## |             Skewness|                -0.09|                  0.7|                 0.56|
## |             Kurtosis|                -0.38|                -0.77|                -1.83|
## |       Uncorrected SS|              4050.52|               7528.9|              2462.89|
## |         Corrected SS|               159.15|               306.29|               177.37|
## |      Coeff Variation|                20.93|                21.51|                31.15|
## |      Std. Error Mean|                 0.87|                 1.52|                 2.98|
## |                Range|                 11.1|                 16.1|                 15.4|
## |  Interquartile Range|                  3.9|                 7.08|                 10.2|
## -----------------------------------------------------------------------------------------

Multiple Variable Statistics

The ds_tidy_stats() function returns summary/descriptive statistics for variables in a data frame/tibble.

ds_tidy_stats(mtcarz, mpg, disp, hp)

## # A tibble: 3 x 16
##   vars    min   max  mean t_mean median  mode range variance  stdev  skew
##   <chr> <dbl> <dbl> <dbl>  <dbl>  <dbl> <dbl> <dbl>    <dbl>  <dbl> <dbl>
## 1 disp   71.1 472   231.   228    196.  276.  401.   15361.  124.   0.420
## 2 hp     52   335   147.   144.   123   110   283     4701.   68.6  0.799
## 3 mpg    10.4  33.9  20.1   20.0   19.2  10.4  23.5     36.3   6.03 0.672
## # ... with 5 more variables: kurtosis <dbl>, coeff_var <dbl>, q1 <dbl>,
## #   q3 <dbl>, iqrange <dbl>

Measures

If you want to view the measure of location, variation, symmetry, percentiles and extreme observations as tibbles, use the below functions. All of them, except for ds_extreme_obs() will work with single or multiple variables. If you do not specify the variables, they will return the results for all the continuous variables in the data set.

Measures of Location

ds_measures_location(mtcarz)

## # A tibble: 6 x 5
##   var     mean trim_mean median   mode
##   <chr>  <dbl>     <dbl>  <dbl>  <dbl>
## 1 disp  231.      228    196.   276.  
## 2 drat    3.60      3.58   3.70   3.07
## 3 hp    147.      144.   123    110   
## 4 mpg    20.1      20.0   19.2   10.4 
## 5 qsec   17.8      17.8   17.7   17.0 
## 6 wt      3.22      3.20   3.32   3.44

Measures of Variation

ds_measures_variation(mtcarz)

## # A tibble: 6 x 7
##   var    range     iqr  variance      sd coeff_var std_error
##   <chr>  <dbl>   <dbl>     <dbl>   <dbl>     <dbl>     <dbl>
## 1 disp  401.   205.    15361.    124.         53.7   21.9   
## 2 drat    2.17   0.840     0.286   0.535      14.9    0.0945
## 3 hp    283     83.5    4701.     68.6        46.7   12.1   
## 4 mpg    23.5    7.38     36.3     6.03       30.0    1.07  
## 5 qsec    8.40   2.01      3.19    1.79       10.0    0.316 
## 6 wt      3.91   1.03      0.957   0.978      30.4    0.173

Measures of Symmetry

ds_measures_symmetry(mtcarz)

## # A tibble: 6 x 3
##   var   skewness kurtosis
##   <chr>    <dbl>    <dbl>
## 1 disp     0.420  -1.07  
## 2 drat     0.293  -0.450 
## 3 hp       0.799   0.275 
## 4 mpg      0.672  -0.0220
## 5 qsec     0.406   0.865 
## 6 wt       0.466   0.417

Percentiles

ds_percentiles(mtcarz)

## # A tibble: 6 x 12
##   var     min  per1  per5 per10     q1 median     q3  per95  per90  per99
##   <chr> <dbl> <dbl> <dbl> <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>
## 1 disp  71.1  72.5  77.4  80.6  121.   196.   326    449    396.   468.  
## 2 drat   2.76  2.76  2.85  3.01   3.08   3.70   3.92   4.31   4.21   4.78
## 3 hp    52    55.1  63.6  66     96.5  123    180    254.   244.   313.  
## 4 mpg   10.4  10.4  12.0  14.3   15.4   19.2   22.8   31.3   30.1   33.4 
## 5 qsec  14.5  14.5  15.0  15.5   16.9   17.7   18.9   20.1   20.0   22.1 
## 6 wt     1.51  1.54  1.74  1.96   2.58   3.32   3.61   5.29   4.05   5.40
## # ... with 1 more variable: max <dbl>

Categorical Data

Cross Tabulation

The ds_cross_table() function creates two way tables of categorical variables.

ds_cross_table(mtcarz, cyl, gear)

##     Cell Contents
##  |---------------|
##  |     Frequency |
##  |       Percent |
##  |       Row Pct |
##  |       Col Pct |
##  |---------------|
## 
##  Total Observations:  32 
## 
## ----------------------------------------------------------------------------
## |              |                           gear                            |
## ----------------------------------------------------------------------------
## |          cyl |            3 |            4 |            5 |    Row Total |
## ----------------------------------------------------------------------------
## |            4 |            1 |            8 |            2 |           11 |
## |              |        0.031 |         0.25 |        0.062 |              |
## |              |         0.09 |         0.73 |         0.18 |         0.34 |
## |              |         0.07 |         0.67 |          0.4 |              |
## ----------------------------------------------------------------------------
## |            6 |            2 |            4 |            1 |            7 |
## |              |        0.062 |        0.125 |        0.031 |              |
## |              |         0.29 |         0.57 |         0.14 |         0.22 |
## |              |         0.13 |         0.33 |          0.2 |              |
## ----------------------------------------------------------------------------
## |            8 |           12 |            0 |            2 |           14 |
## |              |        0.375 |            0 |        0.062 |              |
## |              |         0.86 |            0 |         0.14 |         0.44 |
## |              |          0.8 |            0 |          0.4 |              |
## ----------------------------------------------------------------------------
## | Column Total |           15 |           12 |            5 |           32 |
## |              |        0.468 |        0.375 |        0.155 |              |
## ----------------------------------------------------------------------------

If you want the above result as a tibble, use ds_twoway_table().

ds_twoway_table(mtcarz, cyl, gear)

## Joining, by = c("cyl", "gear", "count")

## # A tibble: 8 x 6
##   cyl   gear  count percent row_percent col_percent
##   <fct> <fct> <int>   <dbl>       <dbl>       <dbl>
## 1 4     3         1  0.0312      0.0909      0.0667
## 2 4     4         8  0.25        0.727       0.667 
## 3 4     5         2  0.0625      0.182       0.4   
## 4 6     3         2  0.0625      0.286       0.133 
## 5 6     4         4  0.125       0.571       0.333 
## 6 6     5         1  0.0312      0.143       0.2   
## 7 8     3        12  0.375       0.857       0.8   
## 8 8     5         2  0.0625      0.143       0.4

A plot() method has been defined which will generate:

Grouped Bar Plots

k <- ds_cross_table(mtcarz, cyl, gear)
plot(k)

Stacked Bar Plots

k <- ds_cross_table(mtcarz, cyl, gear)
plot(k, stacked = TRUE)

Proportional Bar Plots

k <- ds_cross_table(mtcarz, cyl, gear)
plot(k, proportional = TRUE)

Frequency Table

The ds_freq_table() function creates frequency tables.

ds_freq_table(mtcarz, cyl)

##                              Variable: cyl                              
## -----------------------------------------------------------------------
## Levels     Frequency    Cum Frequency       Percent        Cum Percent  
## -----------------------------------------------------------------------
##    4          11             11              34.38            34.38    
## -----------------------------------------------------------------------
##    6           7             18              21.88            56.25    
## -----------------------------------------------------------------------
##    8          14             32              43.75             100     
## -----------------------------------------------------------------------
##  Total        32              -             100.00              -      
## -----------------------------------------------------------------------

A plot() method has been defined which will create a bar plot.

k <- ds_freq_table(mtcarz, cyl)
plot(k)

Multiple One Way Tables

The ds_auto_freq_table() function creates multiple one way tables by creating a frequency table for each categorical variable in a data set. You can also specify a subset of variables if you do not want all the variables in the data set to be used.

ds_auto_freq_table(mtcarz)

##                              Variable: cyl                              
## -----------------------------------------------------------------------
## Levels     Frequency    Cum Frequency       Percent        Cum Percent  
## -----------------------------------------------------------------------
##    4          11             11              34.38            34.38    
## -----------------------------------------------------------------------
##    6           7             18              21.88            56.25    
## -----------------------------------------------------------------------
##    8          14             32              43.75             100     
## -----------------------------------------------------------------------
##  Total        32              -             100.00              -      
## -----------------------------------------------------------------------
## 
##                              Variable: vs                               
## -----------------------------------------------------------------------
## Levels     Frequency    Cum Frequency       Percent        Cum Percent  
## -----------------------------------------------------------------------
##    0          18             18              56.25            56.25    
## -----------------------------------------------------------------------
##    1          14             32              43.75             100     
## -----------------------------------------------------------------------
##  Total        32              -             100.00              -      
## -----------------------------------------------------------------------
## 
##                              Variable: am                               
## -----------------------------------------------------------------------
## Levels     Frequency    Cum Frequency       Percent        Cum Percent  
## -----------------------------------------------------------------------
##    0          19             19              59.38            59.38    
## -----------------------------------------------------------------------
##    1          13             32              40.62             100     
## -----------------------------------------------------------------------
##  Total        32              -             100.00              -      
## -----------------------------------------------------------------------
## 
##                             Variable: gear                              
## -----------------------------------------------------------------------
## Levels     Frequency    Cum Frequency       Percent        Cum Percent  
## -----------------------------------------------------------------------
##    3          15             15              46.88            46.88    
## -----------------------------------------------------------------------
##    4          12             27              37.5             84.38    
## -----------------------------------------------------------------------
##    5           5             32              15.62             100     
## -----------------------------------------------------------------------
##  Total        32              -             100.00              -      
## -----------------------------------------------------------------------
## 
##                             Variable: carb                              
## -----------------------------------------------------------------------
## Levels     Frequency    Cum Frequency       Percent        Cum Percent  
## -----------------------------------------------------------------------
##    1           7              7              21.88            21.88    
## -----------------------------------------------------------------------
##    2          10             17              31.25            53.12    
## -----------------------------------------------------------------------
##    3           3             20              9.38             62.5     
## -----------------------------------------------------------------------
##    4          10             30              31.25            93.75    
## -----------------------------------------------------------------------
##    6           1             31              3.12             96.88    
## -----------------------------------------------------------------------
##    8           1             32              3.12              100     
## -----------------------------------------------------------------------
##  Total        32              -             100.00              -      
## -----------------------------------------------------------------------

Multiple Two Way Tables

The ds_auto_cross_table() function creates multiple two way tables by creating a cross table for each unique pair of categorical variables in a data set. You can also specify a subset of variables if you do not want all the variables in the data set to be used.

ds_auto_cross_table(mtcarz, cyl, gear, am)

##     Cell Contents
##  |---------------|
##  |     Frequency |
##  |       Percent |
##  |       Row Pct |
##  |       Col Pct |
##  |---------------|
## 
##  Total Observations:  32 
## 
##                                 cyl vs gear                                 
## ----------------------------------------------------------------------------
## |              |                           gear                            |
## ----------------------------------------------------------------------------
## |          cyl |            3 |            4 |            5 |    Row Total |
## ----------------------------------------------------------------------------
## |            4 |            1 |            8 |            2 |           11 |
## |              |        0.031 |         0.25 |        0.062 |              |
## |              |         0.09 |         0.73 |         0.18 |         0.34 |
## |              |         0.07 |         0.67 |          0.4 |              |
## ----------------------------------------------------------------------------
## |            6 |            2 |            4 |            1 |            7 |
## |              |        0.062 |        0.125 |        0.031 |              |
## |              |         0.29 |         0.57 |         0.14 |         0.22 |
## |              |         0.13 |         0.33 |          0.2 |              |
## ----------------------------------------------------------------------------
## |            8 |           12 |            0 |            2 |           14 |
## |              |        0.375 |            0 |        0.062 |              |
## |              |         0.86 |            0 |         0.14 |         0.44 |
## |              |          0.8 |            0 |          0.4 |              |
## ----------------------------------------------------------------------------
## | Column Total |           15 |           12 |            5 |           32 |
## |              |        0.468 |        0.375 |        0.155 |              |
## ----------------------------------------------------------------------------
## 
## 
##                          cyl vs am                           
## -------------------------------------------------------------
## |              |                     am                     |
## -------------------------------------------------------------
## |          cyl |            0 |            1 |    Row Total |
## -------------------------------------------------------------
## |            4 |            3 |            8 |           11 |
## |              |        0.094 |         0.25 |              |
## |              |         0.27 |         0.73 |         0.34 |
## |              |         0.16 |         0.62 |              |
## -------------------------------------------------------------
## |            6 |            4 |            3 |            7 |
## |              |        0.125 |        0.094 |              |
## |              |         0.57 |         0.43 |         0.22 |
## |              |         0.21 |         0.23 |              |
## -------------------------------------------------------------
## |            8 |           12 |            2 |           14 |
## |              |        0.375 |        0.062 |              |
## |              |         0.86 |         0.14 |         0.44 |
## |              |         0.63 |         0.15 |              |
## -------------------------------------------------------------
## | Column Total |           19 |           13 |           32 |
## |              |        0.594 |        0.406 |              |
## -------------------------------------------------------------
## 
## 
##                          gear vs am                          
## -------------------------------------------------------------
## |              |                     am                     |
## -------------------------------------------------------------
## |         gear |            0 |            1 |    Row Total |
## -------------------------------------------------------------
## |            3 |           15 |            0 |           15 |
## |              |        0.469 |            0 |              |
## |              |            1 |            0 |         0.47 |
## |              |         0.79 |            0 |              |
## -------------------------------------------------------------
## |            4 |            4 |            8 |           12 |
## |              |        0.125 |         0.25 |              |
## |              |         0.33 |         0.67 |         0.38 |
## |              |         0.21 |         0.62 |              |
## -------------------------------------------------------------
## |            5 |            0 |            5 |            5 |
## |              |            0 |        0.156 |              |
## |              |            0 |            1 |         0.16 |
## |              |            0 |         0.38 |              |
## -------------------------------------------------------------
## | Column Total |           19 |           13 |           32 |
## |              |        0.594 |        0.406 |              |
## -------------------------------------------------------------

Learning More

The descriptr website includes comprehensive documentation on using the package, including the following articles that cover various aspects of using descriptr:

Continuous Data - for summarizing continuous data.
Categorical Data - for summarizing categorical data.
Visualization - for generating different types of plots.

Feedback

All feedback is welcome. Issues (bugs and feature requests) can be posted to github tracker. For help with code or other related questions, feel free to reach me hebbali.aravind@gmail.com.

RFM Analysis in R

Tue, 12 Feb 2019 00:00:00 +0000

I am pleased to announce the rfm package, a set of tools for recency, frequency and monetary value analysis, designed keeping in mind beginner/intermediate R users. It can handle:

transaction level data
customer level data

Installation

# Install release version from CRAN
install.packages("rfm")

# Install development version from GitHub
# install.packages("devtools")
devtools::install_github("rsquaredacademy/rfm")

Shiny App

rfm includes a shiny app which can be launched using

rfm_launch_app()

or try the live version here.

Read on to learn more about the features of rfm, or see the rfm website for detailed documentation on using the package.

Introduction

## Warning: package 'knitr' was built under R version 3.5.2

## Warning: package 'DT' was built under R version 3.5.2

RFM (recency, frequency, monetary) analysis is a behavior based technique used to segment customers by examining their transaction history such as

how recently a customer has purchased (recency)
how often they purchase (frequency)
how much the customer spends (monetary)

It is based on the marketing axiom that 80% of your business comes from 20% of your customers. RFM helps to identify customers who are more likely to respond to promotions by segmenting them into various categories.

Data

To calculate the RFM score for each customer we need transaction data which should include the following:

a unique customer id
date of transaction/order
transaction/order amount

rfm includes a sample data set rfm_data_orders which includes the above details:

rfm_data_orders

## # A tibble: 4,906 x 3
##    customer_id         order_date revenue
##    <chr>               <date>       <dbl>
##  1 Mr. Brion Stark Sr. 2004-12-20      32
##  2 Ethyl Botsford      2005-05-02      36
##  3 Hosteen Jacobi      2004-03-06     116
##  4 Mr. Edw Frami       2006-03-15      99
##  5 Josef Lemke         2006-08-14      76
##  6 Julisa Halvorson    2005-05-28      56
##  7 Judyth Lueilwitz    2005-03-09     108
##  8 Mr. Mekhi Goyette   2005-09-23     183
##  9 Hansford Moen PhD   2005-09-07      30
## 10 Fount Flatley       2006-04-12      13
## # ... with 4,896 more rows

RFM Score

So how is the RFM score computed for each customer? The below steps explain the process:

A recency score is assigned to each customer based on date of most recent purchase. The score is generated by binning the recency values into a number of categories (default is 5). For example, if you use four categories, the customers with the most recent purchase dates receive a recency ranking of 4, and those with purchase dates in the distant past receive a recency ranking of 1.
A frequency ranking is assigned in a similar way. Customers with high purchase frequency are assigned a higher score (4 or 5) and those with lowest frequency are assigned a score 1.
Monetary score is assigned on the basis of the total revenue generated by the customer in the period under consideration for the analysis. Customers with highest revenue/order amount are assigned a higher score while those with lowest revenue are assigned a score of 1.
A fourth score, RFM score is generated which is simply the three individual scores concatenated into a single value.

The customers with the highest RFM scores are most likely to respond to an offer. Now that we have understood how the RFM score is computed, it is time to put it into practice. Use rfm_table_order() to generate the score for each customer from the sample data set rfm_data_orders.

rfm_table_order() takes 8 inputs:

data: a data set with
- unique customer id
- date of transaction
- and amount
customer_id: name of the customer id column
order_date: name of the transaction date column
revenue: name of the transaction amount column
analysis_date: date of analysis
recency_bins: number of rankings for recency score (default is 5)
frequency_bins: number of rankings for frequency score (default is 5)
monetary_bins: number of rankings for monetary score (default is 5)

RFM Table

analysis_date <- lubridate::as_date("2006-12-31", tz = "UTC")
rfm_result <- rfm_table_order(rfm_data_orders, customer_id, order_date, revenue, analysis_date)
rfm_result

customer_id	date_most_recent	recency_days	transaction_count	amount	recency_score	frequency_score	monetary_score	rfm_score
Abbey O’Reilly DVM	2006-06-09	205	6	472	3	4	3	343
Add Senger	2006-08-13	140	3	340	4	1	2	412
Aden Lesch Sr.	2006-06-20	194	4	405	3	2	3	323
Admiral Senger	2006-08-21	132	5	448	4	3	3	433
Agness O’Keefe	2006-10-02	90	9	843	5	5	5	555
Aileen Barton	2006-10-08	84	9	763	5	5	5	555
Ailene Hermann	2006-03-25	281	8	699	3	5	5	355
Aiyanna Bruen PhD	2006-04-29	246	4	157	3	2	1	321
Ala Schmidt DDS	2006-01-16	349	3	363	2	1	2	212
Alannah Borer	2005-04-21	619	4	196	1	2	1	121

rfm_table_order() will return the following columns as seen in the above table:

customer_id: unique customer id
date_most_recent: date of most recent visit
recency_days: days since the most recent visit
transaction_count: number of transactions of the customer
amount: total revenue generated by the customer
recency_score: recency score of the customer
frequency_score: frequency score of the customer
monetary_score: monetary score of the customer
rfm_score: RFM score of the customer

Segments

Let us classify our customers based on the individual recency, frequency and monetary scores.

Segment	Description	R	F	M
Champions	Bought recently, buy often and spend the most	4 - 5	4 - 5	4 - 5
Loyal Customers	Spend good money. Responsive to promotions	2 - 5	3 - 5	3 - 5
Potential Loyalist	Recent customers, spent good amount, bought more than once	3 - 5	1 - 3	1 - 3
New Customers	Bought more recently, but not often	4 - 5	<= 1	<= 1
Promising	Recent shoppers, but haven’t spent much	3 - 4	<= 1	<= 1
Need Attention	Above average recency, frequency & monetary values	2 - 3	2 - 3	2 - 3
About To Sleep	Below average recency, frequency & monetary values	2 - 3	<= 2	<= 2
At Risk	Spent big money, purchased often but long time ago	<= 2	2 - 5	2 - 5
Can’t Lose Them	Made big purchases and often, but long time ago	<= 1	4 - 5	4 - 5
Hibernating	Low spenders, low frequency, purchased long time ago	1 - 2	1 - 2	1 - 2
Lost	Lowest recency, frequency & monetary scores	<= 2	<= 2	<= 2

Segmented Customer Data

We can use the segmented data to identify

best customers
loyal customers
at risk customers
and lost customers

Once we have classified a customer into a particular segment, we can take appropriate action to increase his/her lifetime value.

Segment Size

Now that we have defined and segmented our customers, let us examine the distribution of customers across the segments. Ideally, we should have very few or no customer in segments such as At Risk or Needs Attention.

segments %>%
  count(segment) %>%
  arrange(desc(n)) %>%
  rename(Segment = segment, Count = n)

## # A tibble: 10 x 2
##    Segment            Count
##    <chr>              <int>
##  1 Loyal Customers      288
##  2 Lost                 185
##  3 At Risk              180
##  4 Potential Loyalist   147
##  5 About To Sleep        58
##  6 Others                48
##  7 Need Attention        34
##  8 Promising             21
##  9 Can't Lose Them       17
## 10 New Customers         17

We can also examine the median recency, frequency and monetary value across segments to ensure that the logic used for customer classification is sound and practical.

Median Recency

rfm_plot_median_recency(segments)

Median Frequency

rfm_plot_median_frequency(segments)

Median Monetary Value

rfm_plot_median_monetary(segments)

Heat Map

The heat map shows the average monetary value for different categories of recency and frequency scores. Higher scores of frequency and recency are characterized by higher average monetary value as indicated by the darker areas in the heatmap.

rfm_heatmap(rfm_result)

Bar Chart

Use rfm_bar_chart() to generate the distribution of monetary scores for the different combinations of frequency and recency scores.

rfm_bar_chart(rfm_result)

Histogram

Use rfm_histograms() to examine the relative distribution of

monetary value (total revenue generated by each customer)
recency days (days since the most recent visit for each customer)
frequency (transaction count for each customer)

rfm_histograms(rfm_result)

Customers by Orders

Visualize the distribution of customers across orders.

rfm_order_dist(rfm_result)

Scatter Plots

The best customers are those who:

bought most recently
most often
and spend the most

Now let us examine the relationship between the above.

Recency vs Monetary Value

Customers who visited more recently generated more revenue compared to those who visited in the distant past. The customers who visited in the recent past are more likely to return compared to those who visited long time ago as most of those would be lost customers. As such, higher revenue would be associated with most recent visits.

rfm_rm_plot(rfm_result)

Frequency vs Monetary Value

As the frequency of visits increases, the revenue generated also increases. Customers who visit more frquently are your champion customers, loyal customers or potential loyalists and they drive higher revenue.

rfm_fm_plot(rfm_result)

Recency vs Frequency

Customers with low frequency visited in the distant past while those with high frequency have visited in the recent past. Again, the customers who visited in the recent past are more likely to return compared to those who visited long time ago. As such, higher frequency would be associated with the most recent visits.

rfm_rf_plot(rfm_result)

Learning More

The rfm website includes comprehensive documentation on using the package, including the following articles that cover various aspects of using rfm:

RFM Customer Level Data - shows you how to handle customer level data.
RFM Transaction Level Data - shows you how to handle transaction level data.

Feedback

rfm has been on CRAN for a few months now while we were fixing bugs and making the API stable. All feedback is welcome. Issues (bugs and feature requests) can be posted to github tracker. For help with code or other related questions, feel free to reach me hebbali.aravind@gmail.com.

Introducing olsrr

Thu, 07 Feb 2019 00:00:00 +0000

I am pleased to announce the olsrr package, a set of tools for improved output from linear regression models, designed keeping in mind beginner/intermediate R users. The package includes:

comprehensive regression output
variable selection procedures
heteroskedasticiy, collinearity diagnostics and measures of influence
various plots and underlying data

If you know how to build models using lm(), you will find olsrr very useful. Most of the functions use an object of class lm as input. So you just need to build a model using lm() and then pass it onto the functions in olsrr. Once you have picked up enough knowledge of R, you can move on to more intuitive approach offered by tidymodels etc. as they offer more flexibility, which olsrr does not.

Installation

# Install release version from CRAN
install.packages("olsrr")

# Install development version from GitHub
# install.packages("devtools")
devtools::install_github("rsquaredacademy/olsrr")

Shiny App

olsrr includes a shiny app which can be launched using

ols_launch_app()

or try the live version here.

Read on to learn more about the features of olsrr, or see the olsrr website for detailed documentation on using the package.

Regression Output

model <- lm(mpg ~ disp + hp + wt + qsec, data = mtcars)
ols_regress(model)

##                         Model Summary                          
## --------------------------------------------------------------
## R                       0.914       RMSE                2.622 
## R-Squared               0.835       Coef. Var          13.051 
## Adj. R-Squared          0.811       MSE                 6.875 
## Pred R-Squared          0.771       MAE                 1.858 
## --------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
## 
##                                ANOVA                                 
## --------------------------------------------------------------------
##                 Sum of                                              
##                Squares        DF    Mean Square      F         Sig. 
## --------------------------------------------------------------------
## Regression     940.412         4        235.103    34.195    0.0000 
## Residual       185.635        27          6.875                     
## Total         1126.047        31                                    
## --------------------------------------------------------------------
## 
##                                   Parameter Estimates                                    
## ----------------------------------------------------------------------------------------
##       model      Beta    Std. Error    Std. Beta      t        Sig      lower     upper 
## ----------------------------------------------------------------------------------------
## (Intercept)    27.330         8.639                  3.164    0.004     9.604    45.055 
##        disp     0.003         0.011        0.055     0.248    0.806    -0.019     0.025 
##          hp    -0.019         0.016       -0.212    -1.196    0.242    -0.051     0.013 
##          wt    -4.609         1.266       -0.748    -3.641    0.001    -7.206    -2.012 
##        qsec     0.544         0.466        0.161     1.166    0.254    -0.413     1.501 
## ----------------------------------------------------------------------------------------

In the presence of interaction terms in the model, the predictors are scaled and centered before computing the standardized betas. ols_regress() will detect interaction terms automatically but in case you have created a new variable instead of using the inline function, you can indicate the presence of interaction terms by setting iterm to TRUE.

Residual Diagnostics

olsrr offers tools for detecting violation of standard regression assumptions:

Residual QQ plot
Residual normality test
Residual vs Fitted plot
Residual histogram

ols_plot_resid_qq(model)

See Residual Diagnostics for more details.

Heteroskedasticity

olsrr provides the following 4 tests for detecting heteroscedasticity:

Bartlett Test
Breusch Pagan Test
Score Test
F Test

ols_test_breusch_pagan(model)

## 
##  Breusch Pagan Test for Heteroskedasticity
##  -----------------------------------------
##  Ho: the variance is constant            
##  Ha: the variance is not constant        
## 
##              Data               
##  -------------------------------
##  Response : mpg 
##  Variables: fitted values of mpg 
## 
##         Test Summary         
##  ----------------------------
##  DF            =    1 
##  Chi2          =    0.5884673 
##  Prob > Chi2   =    0.4430124

See Heteroskedasticity for more details.

Collinearity Diagnostics

VIF, Tolerance and condition indices to detect collinearity and plots for assessing mode fit and contributions of variables.

ols_coll_diag(model)

## Tolerance and Variance Inflation Factor
## ---------------------------------------
## # A tibble: 4 x 3
##   Variables Tolerance   VIF
##   <chr>         <dbl> <dbl>
## 1 disp          0.125  7.99
## 2 hp            0.194  5.17
## 3 wt            0.145  6.92
## 4 qsec          0.319  3.13
## 
## 
## Eigenvalue and Condition Index
## ------------------------------
##    Eigenvalue Condition Index   intercept        disp          hp
## 1 4.721487187        1.000000 0.000123237 0.001132468 0.001413094
## 2 0.216562203        4.669260 0.002617424 0.036811051 0.027751289
## 3 0.050416837        9.677242 0.001656551 0.120881424 0.392366164
## 4 0.010104757       21.616057 0.025805998 0.777260487 0.059594623
## 5 0.001429017       57.480524 0.969796790 0.063914571 0.518874831
##             wt         qsec
## 1 0.0005253393 0.0001277169
## 2 0.0002096014 0.0046789491
## 3 0.0377028008 0.0001952599
## 4 0.7017528428 0.0024577686
## 5 0.2598094157 0.9925403056

See Collinearity Diagnostics for more details.

Measures of Influence

olsrr offers the following tools to detect influential observations:

Cook’s D Bar Plot
Cook’s D Chart
DFBETAs Panel
DFFITs Plot
Studentized Residual Plot
Standardized Residual Chart
Studentized Residuals vs Leverage Plot
Deleted Studentized Residual vs Fitted Values Plot
Hadi Plot
Potential Residual Plot

ols_plot_resid_lev(model)

See Measures of Influence for more details.

Variable Selection

Different variable selection procedures such as all possible regression, best subset regression, stepwise regression, stepwise forward regression and stepwise backward regression.

model <- lm(y ~ ., data = stepdata)
ols_step_both_aic(model)

## Stepwise Selection Method 
## -------------------------
## 
## Candidate Terms: 
## 
## 1 . x1 
## 2 . x2 
## 3 . x3 
## 4 . x4 
## 5 . x5 
## 6 . x6 
## 
## 
## Variables Entered/Removed: 
## 
## - x6 added 
## - x1 added 
## - x3 added 
## - x2 added 
## - x6 removed 
## - x4 added 
## 
## No more variables to be added or removed.

## 
## 
##                                   Stepwise Summary                                  
## ----------------------------------------------------------------------------------
## Variable     Method        AIC         RSS        Sum Sq       R-Sq      Adj. R-Sq 
## ----------------------------------------------------------------------------------
## x6          addition    33473.297    6241.497    13986.736    0.69145      0.69143 
## x1          addition    32931.758    6074.156    14154.076    0.69972      0.69969 
## x3          addition    31912.722    5771.842    14456.391    0.71466      0.71462 
## x2          addition    29304.296    5065.587    15162.646    0.74958      0.74953 
## x6          removal     29302.317    5065.592    15162.641    0.74958      0.74954 
## x4          addition    29300.814    5064.705    15163.528    0.74962      0.74957 
## ----------------------------------------------------------------------------------

See Variable Selection for more details.

Learning More

The olsrr website includes comprehensive documentation on using the package, including the following articles that cover various aspects of using olsrr:

Variable Selection - Different variable selection procedures such as all possible regression, best subset regression, stepwise regression, stepwise forward regression and stepwise backward regression.
Residual Diagnostics - Includes plots to examine residuals to validate OLS assumptions.
Heteroskedasticity - Tests for heteroskedasticity include bartlett test, breusch pagan test, score test and f test.
Collinearity Diagnostics - VIF, Tolerance and condition indices to detect collinearity and plots for assessing mode fit and contributions of variables.
Measures of Influence - Includes 10 different plots to detect and identify influential observations.

Feedback

olsrr has been on CRAN for more than an year while we were fixing bugs and making the API stable. All feedback is welcome. Issues (bugs and feature requests) can be posted to github tracker. For help with code or other related questions, feel free to reach me hebbali.aravind@gmail.com.

Mon, 01 Jan 0001 00:00:00 +0000

news

Mon, 01 Jan 0001 00:00:00 +0000

readme

Mon, 01 Jan 0001 00:00:00 +0000

Untitled

Mon, 01 Jan 0001 00:00:00 +0000

ls