Generalized Linear Models (GLM) including logistic regression with disk.frame

suppressPackageStartupMessages(library(disk.frame))

if(interactive()) {
  setup_disk.frame() 
} else {
  # only use 1 work to pass CRAN check
  setup_disk.frame(1)
}
#> The number of workers available for disk.frame is 1

GLMs

Prerequisites

In this article, we will assume you are familiar with Generalized Linear Models (GLMs). You are also expected to have basic working knowledge of {disk.frame}, see this {disk.frame} Quick Start.

Introduction

One can fit a GLM using the glm function. For example,

would fit a linear model on the data cars with dist as the target and speed as the explanatory variable. You can inspect the results of the model fit using

or if you have {broom} installed

With {disk.frame}, you can run GLM dfglm function, where the df stands for disk.frame of course!

The syntax didn’t change at all! You are able to enjoy the benefits of disk.frame when dealing with larger-than-RAM data.

Logistic regression

Logistic regression is one of the most commonly deployed machine learning (ML) models. It is often used to build binary classification models

The arguments to the dfglm function are the same as the arguments to biglm::bigglm which are based on the glm function. Please check their documentations for other argument options.

Notes

{disk.frame} uses {biglm} and {speedglm} as the backend for GLMs. Unfortunately, neither package is managed on open-source platforms, so it’s more difficult to contribute to them by making bug fixes and submitting bug reports. So bugs are likely to persists. There is an active effort on disk.frame to look for alternatives. Example of avenues to explore include tighter integration with {keras}, h2o, or Julia’s OnlineStats.jl for model fit purposes.

Another package for larger-than-RAM glm fitting, {bigFastlm}, has been taken off CRAN, it is managed on Github.

Currently, parallel processing of GLM fit are not possible with {disk.frame}.