Using {disk.frame} for Large Datasets {unnumbered}

disk.frame is an R package designed for handling large datasets that exceed available RAM by storing data on disk in a chunked format. It allows users to perform data manipulation using familiar dplyr syntax while processing data in parallel across multiple CPU cores. This tutorial demonstrates how to use disk.frame to work with the NYC taxi dataset, showcasing its capabilities for out-of-memory data manipulation. Note that disk.frame is soft-deprecated, and users are encouraged to consider the arrow package for new projects, but this tutorial focuses on disk.frame for educational purposes.

disk.frame Installation Issues and Solutions

When attempting to install the disk.frame R package, users may encounter installation errors, especially on newer versions of R or certain operating systems. A common issue involves failed dependency installations (such as fst or bit64), or incompatibilities with recent CRAN policies. In some cases, disk.frame’s dependencies require compilation with specific system libraries, or users may need to install an older version of the package that matches their R environment.

To fix these installation problems: - Ensure all system dependencies required for compiling R packages are installed (for example, build tools on Windows or development libraries on Linux). - Explicitly install any missing dependencies (fst, bit64, dplyr, future) before installing disk.frame. - If a direct installation from CRAN fails, consider installing the development version from GitHub, which may include fixes for compatibility issues:

# Install or update build tools if necessary (on Windows)
# install.packages("Rtools")

# Install disk.frame and dependencies from CRAN (preferred)
install.packages(c("fst", "bit64", "dplyr", "future")) # Install dependencies first
install.packages("disk.frame")

# If CRAN version fails, install the latest version from GitHub:
# install.packages("devtools")
# devtools::install_github("xiaodaigh/disk.frame")

With these steps, the disk.frame package should install successfully, allowing you to process large datasets efficiently in R. In this tutorial, we’ll use disk.frame (properly installed and loaded as shown above) to work with the NYC taxi dataset, demonstrating out-of-memory data manipulation using familiar dplyr and data.table-like syntax.

Check and Install Required R Packages

Code 
packages <- c(
          'tidyverse',
          'data.table',
          'arrow',
          'disk.frame',
          'fst',
          'future'
          
)
Code 
# Install missing packages
new_packages <- packages[!(packages %in% installed.packages()[,"Package"])]
if(length(new_packages)) install.packages(new_packages)

Verify Installation

Code 
# Verify installation
cat("Installed packages:\n")
Installed packages:
Code 
print(sapply(packages, requireNamespace, quietly = TRUE))
 tidyverse data.table      arrow disk.frame        fst     future 
      TRUE       TRUE       TRUE       TRUE       TRUE       TRUE

s ### Load Packages

Code 
# Load packages with suppressed messages
loaded_packages <- lapply(packages, function(pkg) {
  if (requireNamespace(pkg, quietly = TRUE)) {
    suppressPackageStartupMessages(library(pkg, character.only = TRUE))
    TRUE
  } else {
    FALSE
  }
})
names(loaded_packages) <- packages

# If disk.frame is unavailable (it is soft-deprecated and may be missing),
# skip executable chunks so the tutorial still renders.
has_disk_frame <- isTRUE(loaded_packages[["disk.frame"]])
if (!has_disk_frame) {
  message("disk.frame is not available in this environment; skipping executable chunks.")
  knitr::opts_chunk$set(eval = FALSE)
}

Check Loaded Packages

Code 
# Check loaded packages
cat("Successfully loaded packages:\n")
Successfully loaded packages:
Code 
print(search()[grepl("package:", search())])
 [1] "package:future"     "package:fst"        "package:disk.frame"
 [4] "package:arrow"      "package:data.table" "package:lubridate" 
 [7] "package:forcats"    "package:stringr"    "package:dplyr"     
[10] "package:purrr"      "package:readr"      "package:tidyr"     
[13] "package:tibble"     "package:ggplot2"    "package:tidyverse" 
[16] "package:stats"      "package:graphics"   "package:grDevices" 
[19] "package:utils"      "package:datasets"   "package:methods"   
[22] "package:base"

Data

We’ll use the 2023 Yellow Taxi Trip Data from NYC Open Data, which contains ~38 million rows with columns like tpep_pickup_datetime, passenger_count, trip_distance, fare_amount, PULocationID, payment_type, and more. We’ll load a subset (January 2023) to manage memory, using {arrow} to read the Parquet file and convert to {data.table}.

Set Up disk.frame

Configure disk.frame to optimize performance. Set the number of chunks based on CPU cores and enable parallel processing with future.

Code 
# Set up disk.frame
setup_disk.frame()
The number of workers available for disk.frame is 40

Set number of chunks (e.g., 2x CPU cores for efficiency)

Code 
ncores <- parallel::detectCores()
options(future.globals.maxSize = Inf) # Allow large data transfers
nchunks_optimal <- 2 * ncores
cat("Optimal number of chunks set to:",nchunks_optimal, "\n")
Optimal number of chunks set to: 160

Define data folder

Code 
dataFolder <- "/home/zia207/WebSites/R_Website/High_Performance_Computing_R/R_Markdown/Data/"

Load Data into disk.frame

Convert the CSV to a disk.frame object, which splits the data into fst files stored on disk. Use csv_to_disk.frame to read yellow_tripdata_2023.csv.

Code 
# Define output folder for disk.frame chunks
df_path <- file.path(tempdir(), "yellow_tripdata_df")

# Convert CSV to disk.frame
df <- csv_to_disk.frame(
  infile = paste0(dataFolder, "yellow_tripdata_2023_01.csv"),
  outdir = df_path,
  header = TRUE,
  nchunks = 2 * ncores, # Number of chunks (adjust based on RAM)
  backend = "data.table" # Use data.table for fast reading
)

# Check disk.frame structure
print(df)
path: "/tmp/Rtmp0gS1rM/yellow_tripdata_df"
nchunks: 160
nrow (at source): 2913955
ncol (at source): 19

This creates a folder (df_path) containing chunked fst files, each representing a subset of the data.

Basic Operations with dplyr Syntax

{disk.frame} supports {dplyr} verbs for data manipulation. Below, filter trips with a fare amount greater than $10 and select key columns.

Filter and select

Code 
# Filter and select
filtered_df <- df %>%
  filter(fare_amount > 10) %>%
  select(VendorID, tpep_pickup_datetime, fare_amount, passenger_count)

# View first few rows (collects to memory, use sparingly)
filtered_df %>% collect() %>% head()
   VendorID tpep_pickup_datetime fare_amount passenger_count
      <int>               <POSc>       <num>           <int>
1:        1  2023-02-01 00:29:33        70.9               0
2:        2  2023-02-01 00:12:28        17.0               1
3:        1  2023-02-01 00:52:40        21.9               1
4:        1  2023-02-01 00:12:39        41.5               1
5:        2  2023-02-01 00:20:40        30.3               1
6:        2  2023-02-01 00:00:28        23.3               1

The collect() function brings results into memory, so use it only for small outputs.

Group-By and Aggregation

Perform aggregations, like calculating average fare by passenger count. disk.frame processes chunks in parallel.

Code 
# Group by passenger_count and calculate mean fare
agg_df <- filtered_df %>%
  group_by(passenger_count) %>%
  summarise(avg_fare = mean(fare_amount, na.rm = TRUE))

# Collect results
agg_df %>% collect()
Warning in collect.summarized_disk.frame(.): These columns that appear in the
group-by and summarise does not appear in the original data set: mean. This set
of action is too hard for disk.frame to figure out the `srckeep` automatically,
you must do the `srckeep` manually.
# A tibble: 11 × 2
   passenger_count avg_fare
             <int>    <dbl>
 1               0     22.4
 2               1     23.7
 3               2     26.6
 4               3     26.0
 5               4     27.6
 6               5     23.4
 7               6     23.8
 8               7     72.6
 9               8     83.2
10               9     92.5
11              NA     21.4

Joining disk.frames

Join two disk.frame objects (e.g., filter subsets and join). Here, we create a subset for short trips and join with the filtered data.

Code 
# Create a subset for trips with distance < 2 miles
short_trips_df <- df %>%
  filter(trip_distance < 2) %>%
  select(VendorID, tpep_pickup_datetime, trip_distance)

# Join with filtered_df
joined_df <- left_join(
  filtered_df,
  short_trips_df,
  by = c("VendorID", "tpep_pickup_datetime")
)

# Collect a sample
joined_df %>% collect() %>% head()
   VendorID tpep_pickup_datetime fare_amount.x passenger_count.x fare_amount.y
      <int>               <POSc>         <num>             <int>         <num>
1:        2  2023-02-01 00:44:17          16.3                 2          16.3
2:        2  2023-02-01 01:29:48          10.7                 1          10.7
3:        2  2023-02-01 01:55:29          21.2                 1          21.2
4:        2  2023-02-01 01:07:12          13.5                 1          13.5
5:        2  2023-02-01 02:42:13          10.7                 1          10.7
6:        2  2023-02-01 02:38:09          14.9                 3          14.9
   passenger_count.y
               <int>
1:                 2
2:                 1
3:                 1
4:                 1
5:                 1
6:                 3

Optimizing Performance

Use srckeep to load only necessary columns, reducing disk I/O. Example: keep only fare_amount and passenger_count.

Code 
# Optimize by keeping only needed columns
df_opt <- df %>% srckeep(c("fare_amount", "passenger_count"))

# Verify columns
colnames(df_opt)
 [1] "VendorID"              "tpep_pickup_datetime"  "tpep_dropoff_datetime"
 [4] "passenger_count"       "trip_distance"         "RatecodeID"           
 [7] "store_and_fwd_flag"    "PULocationID"          "DOLocationID"         
[10] "payment_type"          "fare_amount"           "extra"                
[13] "mta_tax"               "tip_amount"            "tolls_amount"         
[16] "improvement_surcharge" "total_amount"          "congestion_surcharge" 
[19] "airport_fee"

Export Results

Export results to a CSV or convert to arrow for further use (recommended, as disk.frame is soft-deprecated).

Code 
# export to CSV
output_csv <- file.path(tempdir(), "filtered_taxi_data.csv")
write_disk.frame(filtered_df, output_csv, overwrite = TRUE)
path: "/tmp/Rtmp0gS1rM/filtered_taxi_data.csv"
nchunks: 160
nrow (at source): 1900287
ncol (at source): 4
Code 
cat("Filtered data exported to:", output_csv, "\n")
Filtered data exported to: /tmp/Rtmp0gS1rM/filtered_taxi_data.csv

Clean Up

Remove temporary chunk files to free disk space.

Code 
# Delete disk.frame folder
unlink(df_path, recursive = TRUE)

Key Notes

  • disk.frame is ideal for datasets (10-100 GB) too large for RAM but manageable on a single machine.
  • Use chunk_size to balance memory and performance (smaller chunks for low RAM, larger for faster processing). d
  • isk.frame is soft-deprecated; consider arrow for new projects (use as_arrow_dataset for conversion).
  • Monitor disk space, as chunked fst files can grow large.
  • Parallel processing requires sufficient CPU cores and proper future setup.

Summary and Conclusion

This tutorial demonstrated using the {disk.frame} package in R to handle large datasets that exceed available RAM. By chunking data into fst files on disk, {disk.frame} enables efficient processing with familiar dplyr syntax. We covered setup, loading data, filtering, aggregation, joining, performance optimization, and exporting results. While {disk.frame} is useful for medium-sized datasets, it has been soft-deprecated in favor of the {arrow} package for new projects. Users should consider their specific needs and dataset sizes when choosing between these tools.

References

  1. ZJ D (2022). disk.frame: Larger-than-RAM Disk-Based Data Manipulation Framework. R package version 0.7.1, https://diskframe.com.

  2. Wickham H, François R, Henry L, Müller K (2023). dplyr: A Grammar of Data Manipulation. R package version 1.1.2, https://CRAN.R-project.org/package=dplyr.