Code Monkey home page Code Monkey logo

nara's Introduction

nara

R-CMD-check

{nara} provides tools for working with R’s nativeRaster image format to enable fast double-buffered graphics rendering.

Why?

nativeRaster buffers are fast enough to use for rendering at speed >30 frames-per-second.
This makes them useful for games and other interactive applications.

Details

{nara}:

  • is an off-screen rendering buffer.
  • is fast to render.
  • uses in-place operations to avoid memory allocations.
  • is focussed on rendering discrete pixels, so all dimensions are rounded to integer values prior to rendering.
  • some basic drawing primitives are included

What is a nativeRaster and why is it fast?

A nativeRaster is a built-in datatype in R.

It is an integer matrix where each integer represents the RGBA color at a single pixel. The 32-bit integer at each location is interpreted within R to be four color channels (RGBA) represented by 8 bits each.

This way of encoding color information is closer to the internal representation used by graphics devices, and therefore can be faster to render, save and load (as fewer data conversion steps are needed).

Native rasters do not use pre-multiplied alpha.

In-place operation

{nara} is targeted at fast rendering (>30fps), and tries to minimise R function calls and memory allocations.

When updating nativeRaster objects with this package, all changes are done in place on the current object i.e. a new object is not created.

Anti-aliasing/Interpolation

No anti-aliasing is done by the draw methods in this package.

No interpolation is done - x and y values for drawing coordinates are converted to integers.

Installation

You can install from GitHub with:

# install.package('remotes')
remotes::install_github('coolbutuseless/nara')

Vignettes

Static Rendering: Example

The following is a rendering of a single scene with multiple elements.

The interesting thing about this scene that drawing all the objects into the nativeRaster image and rendering to screen can take as little as 5 millseconds.

This means that this scene could render at around 200 frames-per-second.

library(grid)
library(nara)
set.seed(1)

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Create 'nr' image
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
w  <- 10
h  <- 8
nr <- nr_new(w * 30, h * 30, fill = 'grey98')

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Draw a grid of squares
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
colors <- viridisLite::inferno(w * h)
coords  <- expand.grid(y = seq(0, h-1) * 30 + 1, x = seq(0, w-1) * 30 + 1)
nr_rect(nr, x = coords$x, y = coords$y, w = 27, h = 27, fill = colors)

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Draw a bunch of deer sprites
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
nr_blit_list(nr, x = sample(300, 15), y = sample(200, 15), src_list = deer_sprites, src_idx = 1)


#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Add an image read from file (with alpha transparency)
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
img <- png::readPNG(system.file("img", "Rlogo.png", package="png"), native = TRUE)
nr_blit(nr, 0, 0, img)

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Add a polygon
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
thetas <- seq(pi/6, 2*pi, pi/3)
x <- 50 * cos(thetas) + 240
y <- 50 * sin(thetas) + 180
nr_polygon(nr, x = x, y = y, fill = '#556688c0', color = 'blue')

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Add text to the image
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
nr_text_basic(nr, x = 180, y = 20, str = "Hello #RStats", fontsize = 16)

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Copy image to the device
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
grid.raster(nr, interpolate = FALSE)

Static Rendering: Displaying Sprites

Included with {nara} are 16 frames of an animated deer character - see deer_sprites data.

Blit the first deer frame onto a native raster canvas.

library(grid)

nr <- nr_new(100, 32, 'grey80')
nr_blit_list(nr, 2, 0, src_list = deer_sprites, src_idx = 1)
grid.raster(nr, interpolate = FALSE)

Dynamic (realtime) Rendering: Animated deer

The reason to use {nara} is that operations are fast enough that nativeRaster can be used as an in-memory buffer for a double-bufferred rendering system.

Double-buffered rendering is where two buffers are used for rendering with one buffer being shown to the user, and the other existing in memory as a place to render.

In this example, the deer sprite is rendered to a nativeRaster image. This in-memory buffer is then displayed to the user using grid.raster().

By altering the position and animation frame every time the kind is shown, smooth animation is possible.

This simple code runs at well over 100 frames-per-second.

It is unlikely your screen will refresh this fast, but it does indicate that there is plenty of headroom for more complicated computations for each frame.

library(grid)

# Setup a fast graphics device that can render quickly
x11(type = 'cairo', antialias = 'none')
dev.control('inhibit')

# Create the in-memory nativeRaster canvas
nr <- nr_new(100, 32, 'grey80')

# Clear, blit and render => animation!
for (i in -30:110) {
  nr_fill(nr, 'grey80')                    # Clear the nativeRaster
  sprite_idx <- floor((i/3) %% 5) + 11
  nr_blit_list(nr, i, 0, deer_sprites, sprite_idx) # copy deer to nativeRaster
  dev.hold()
  grid.raster(nr, interpolate = FALSE)     # copy nativeRaster to screen
  dev.flush()
  Sys.sleep(0.03)                          # Stop animation running too fast.
}

Live screen recording

Multi-Ball

You can quickly blit (i.e. copy) a sprite into multiple locations on the nativeraster with nr_blit() and nr_blit_list()

In this example 100 random positions and velocities are first created. A character sprite is then blitted to each of these 100 locations.

The positions are updated using the velocities, and the next frame is rendered. In this way multiple sprites are rendered and animated on screen.

library(grid)

# Setup a fast graphics device that can render quickly
x11(type = 'dbcairo', antialias = 'none', width = 8, height = 6)
dev.control('inhibit')

# Number of sprites
N <- 100

# Canvas size
w <- 400 
h <- 300 

# location and movement vector of all the sprites
x  <- sample(w, N, replace = TRUE)
y  <- sample(h, N, replace = TRUE)
vx <- runif(N, 1, 5)


# Create an empty nativeraster with a grey background
nr <- nr_new(w, h, 'white')


for (frame in 1:1000) {
  # Clear the nativeraster and blit in all the deer
  nr_fill(nr, 'white') 
  nr_blit_list(nr, x, y, deer_sprites, floor((frame/3) %% 5 + 11))
  
  # Draw the nativeraster to screen
  dev.hold()
  grid.raster(nr, interpolate = FALSE)
  dev.flush()

  # Update the position of each deer. 
  # Position wraps around
  x <- x + vx
  x <- ifelse(x > w , -32, x)

  # slight pause. Otherwise everything runs too fast!
  Sys.sleep(0.03)
}

Live screen recording

Coordinate System

The coordinate system for nara nativeRaster objects has its origins at the top left corner of the image with coordinates (0, 0).

This is equivalent to {grid} graphics using native units.

It is also how {magick} represents image coordinates, as well as the majority of C graphics libraries.

nara's People

Contributors

coolbutuseless avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

nara's Issues

Shiny integration

Simple question, but can this be integrated in a shiny application?

Optimized polygon rendering

A better optimized polygon rendering algorithm would be nice to have, but is not a priority for the initial release.

Optimizations:

  • Use an active edges list
  • Use Bresenham-style position updating to efficiently update the polygon edges at every new scanline.
  • polygon clipping? Not sure if "juice is worth the squeeze". Does the time spent clipping save enough compared to when it comes time to actually render?

Probably best to start this as a new function in C, and just leave the old one for reference.

References:

Allow SDL surface (with or without OpenGL) instead of x11 graphics device for increased performance?

Thanks for the nice package! (I had been looking for a solution to get fast 2D graphics in R for ages, https://stackoverflow.com/questions/48119360/performant-2d-opengl-graphics-in-r-for-fast-display-of-raster-image-using-rdynca, and this one works for me)
I was just wondering if a further speed increase could be had by using an SDL graphics surface (with or without OpenGL) instead of an x11 graphics device? An SDL graphics window can be created by making the required SDL calls using library(rdyncall) (work is currently in progress to get that back on CRAN, but the archived version still works perfectly well & has some nice SDL 1.2 & OpenGL demos included). See here for a nice demo: https://dyncall.org/demos/soulsalicious/index.html. I presume that moving around 2D sprites presumably would also be faster if SDL and/or OpenGL calls were used? It would also have the advantage that full screen display instead of display in a window could be supported...

`colour_to_integer()` for `"transparent"` and `NA_character_` "colors"

I'm observing that colour_to_integer() is casting "transparent" to an opaque color equivalent to "white" (with a message saying unknown color) and is casting NA_character_ to "#00000000" which is a fully transparent value but a slightly different fully transparent value from what col2rgb() assigns to it. I'd pedantically "expect" colour_to_integer(NA_character_) to instead return 16777215L (the same as colour_to_integer("#FFFFFF00")).

> library("nara")
> colour_to_integer("#FFFFFF00") |> integer_to_colour()
[1] "#ffffff00"
> colour_to_integer("transparent") |> integer_to_colour()
Unknown colour: transparent
[1] "#ffffffff"
> colour_to_integer(NA_character_) |> integer_to_colour()
[1] "#00000000"
> col2rgb(c("transparent", "#FFFFFF00", NA_character_, "white"),  
          alpha = TRUE)
      [,1] [,2] [,3] [,4]
red    255  255  255  255
green  255  255  255  255
blue   255  255  255  255
alpha    0    0    0  255

I've observed that farver::encode_native() also has trouble with these two edge cases: thomasp85/farver#47

CRAN preparation

Done

Bugs

  • None known

Misc fixes

  • Include a "nara.h" header within some boilerplate for operating on nativeRasters in C. For use with \code{callme}
  • switch mp4 output to use {fastpng} and {av} directly, rather than going via {magick}
  • Figure out optimal way to render a native raster onto a ggplot.
  • Set a current masked area for rendering. nr_copy_with_mask()
  • Move nr_blit_grid() to C?
  • nr_bezier() ?
  • Need snappier names for str_cols_to_packed_cols and packed_cols_to_hex_cols.
    • encode_native(), col2native(), col2packedint(), col2int(), color_to_native()
  • nr_triangle() ?permission to reproduce / use.
  • Make a better looking example for the first image.
  • Vignette on saving image sequence as gif animation (by converting nativeRasters to magick images)
  • Convert a floating point numeric matrix to nativeRaster using a palette lookup.
  • Align argument names with other packages in the naraverse
  • Vignettes. Or maybe put in narademos repository.
    • spritesheets
    • parallex scrolling
    • boids via R6
    • Setting up a game board using a sprite sheet
    • blit examples. blit vs blit-list vs blit-grid.

Update other packages

Future

  • Expose some C functions for use by other packages?

CRAN

  • desc::desc_normalize()
  • usethis::use_tidy_description()
  • rhub checks: clang-asan, valgrind, rchk
  • winbuilder checks
  • macbuilder checks
  • update copyright info
  • checkhelper::find_missing_tags()
  • checkhelper::check_as_cran()
  • urlchecker::url_check()
  • Check all exported functions have @examples
  • compiler with -Wconversion

Performance improvement of specific use case: displaying a grid of integer values

Hi Mike,

Thanks for developing such a really nice package.

I'd like to ask if you think we could make this use case of mine more performant.

I'd like to use {nara} to display the simulation of cellular automata in real-time.

Here is a toy example:

library(nara)
n_colours <- 5
my_palette <- hsv(seq(0, 1, length.out = n_colours))
w <- 100
h <- 100


for (i in 1:20) {
  m <-
    matrix(data = sample(seq_len(n_colours), size = w * h, replace = TRUE),
           nrow = h)
  m2 <- matrix(my_palette[m], nrow = h)
  nr <- raster_to_nr(m2)
  grid.raster(nr, interpolate = FALSE)
  Sys.sleep(0.1)
}

Assume that m holds the state of the cellular automaton, i.e. an integer matrix that comprises only of positive integer values, starting at 1. For illustrative purposes I'm randomly generating these values with sample(). Assume also that I have a color palette that maps each integer state value to a colour (as provided by the vector my_palette).

Would there be a more performant way (a shortcut, or perhaps a solution based on C code) to go from my matrix m to the nativeRaster object nr that would obviate the calls:

  1. m2 <- matrix(my_palette[m], nrow = h)
  2. nr <- raster_to_nr(m2)

Thanks once again for developing such nice projects!

is 4 channel mandatory?

in my first implementation reading nativeRaster from spatial sources I made 3-channel the default.

but

Error in nr_line(im[[1]], segs[, 1], segs[, 2], segs[, 3], segs[, 4]) : 
  Object is not a nativeRaster with 4 channels

Can this be relaxed to allow 3 channels as well?

(Maybe I shouldn't have done this, but the code for 3 or 4 is in R)

x2 speedup in colour_to_integer mapping when replacing nara::colour_to_integer() with farver::encode_native()

Thanks for letting me discover nativeRaster objects through your nara package. They are blazingly fast!

Just in case you are looking for further optimizations, my benchmark on nara::colour_to_integer() with both a 512 long vector and a 10 million vector shows that farver::encode_native() is close to twice as fast.

with a 10 million long vector:

# Generate a vector of colours:
ramp <- scales::colour_ramp(c("red", "blue"))
large_colormap <- ramp(runif(1E7))
head(large_colormap)
#> [1] "#CA0087" "#EA004B" "#F90021" "#E00060" "#D60074" "#BE0099"
# Benchmark
bench::mark(farver = {farver::encode_native(large_colormap)}, nara = {nara::colour_to_integer(large_colormap)})
#> # A tibble: 2 × 6
#>   expression      min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 farver        284ms    284ms      3.53    38.2MB     3.53
#> 2 nara          603ms    603ms      1.66    38.2MB     0

With a 512 long vector, similar behaviour:

ramp <- scales::colour_ramp(c("red", "blue"))
small_colormap <- ramp(runif(512))
head(small_colormap)
#> [1] "#DC0069" "#8E00CD" "#D30078" "#F70027" "#D50076" "#4800F4"
bench::mark(farver = {farver::encode_native(small_colormap)}, nara = {nara::colour_to_integer(small_colormap)})
#> # A tibble: 2 × 6
#>   expression      min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 farver       13.3µs   15.1µs    63080.    8.53KB        0
#> 2 nara         27.4µs   29.8µs    31544.   50.84KB        0

Created on 2022-09-06 by the reprex package (v2.0.1)

CRAN submission

Hi,

I'm interested in using this for my package R Warrior in the future. Do you plan a CRAN release? That will be required as I plan to have CRAN as the primary distribution method.

Thanks,
Rick (Also from Australia)

Timings to display 1920x1080 nativeRaster slower than to display regular raster using grid.raster or plot.raster?

Based on my question here
https://stackoverflow.com/questions/48119360/performant-2d-opengl-graphics-in-r-for-fast-display-of-raster-image-using-rdynca

I did some timings to display a 1920x1080 raster image, either using a regular raster or using a nara::nativeRaster.

Strangely enough, I get that a regular raster is faster, presumably due to a bottleneck in the raster_to_nr naster to native raster conversion. Do you see what I might be doing wrong?
Other slight problem seems to be that grid.raster(nara::raster_to_nr(rast), interpolate=FALSE) rotates my raster. Any easy solution for that?

# some example data & desired colour mapping of [0-1] ranged data matrix
library(RColorBrewer)
ncol=1080
cols=colorRampPalette(RColorBrewer::brewer.pal(11, "RdYlBu"))(ncol)
colfun=colorRamp(RColorBrewer::brewer.pal(11, "RdYlBu"))
col = rgb(colfun(seq(0,1, length.out = ncol)), max = 255)
mat=matrix(seq(1:1080)/1080,nrow=1920,ncol=1080,byrow=TRUE)
# function to convert matrix of values to colour raster based on given colour mapping
mat2rast = function(mat, col) {
  idx = findInterval(mat, seq(0, 1, length.out = length(col)))
  colors = col[idx]
  rastmat = t(matrix(colors, ncol = ncol(mat), nrow = nrow(mat), byrow = TRUE))
  class(rastmat) = "raster"
  return(rastmat)
}
system.time(rast <- mat2rast(mat, col)) # 0.05s

# Setup a fast graphics device that can render quickly
x11(type = 'dbcairo', antialias = 'none', width = 8, height = 6)

# grid graphics - fastest of the regular solutions
library(grid)
system.time(grid.raster(rast, interpolate=FALSE)) # 0.04s

# plot.raster method - tie with grid.raster?
par(mar=c(0, 0, 0, 0))
system.time(plot(rast, asp=NA)) # 0.04s

# base R image()
par(mar=c(0, 0, 0, 0))
system.time(image(mat,axes=FALSE,useRaster=TRUE,col=cols)) # 0.21s # note Y is flipped to compared to 2 options above - but not so important as I can fill matrix the way I want

# ggplot2 - just for the record...
df=expand.grid(y=1:1080,x=1:1920)
df$z=seq(1,1080)/1080
library(ggplot2)
system.time({q <- qplot(data=df,x=x,y=y,fill=z,geom="raster") + 
  scale_x_continuous(expand = c(0,0)) + 
  scale_y_continuous(expand = c(0,0)) +
  scale_fill_gradientn(colours = cols) + 
  theme_void() + theme(legend.position="none"); print(q)}) # 2.72s

# timings native raster - 2x slower than grid.raster or plot.raster??
system.time(grid.raster(nara::raster_to_nr(rast))) # 0.1s

Movement in first example is jerky for me

On my Windows system the movement of the first example was jerky for me - it would only actually display a frame every second or so... To fix that, I had to insert a dev.hold() and dev.flush() call in the for loop as below...

library(grid)

# Setup a fast graphics device that can render quickly
x11(type = 'dbcairo', antialias = 'none')
dev.control('inhibit')

# Create the in-memory nativeRaster canvas
nr <- nr_new(100, 30, 'grey80')

# Clear, blit and render => animation!
for (i in -30:110) { 
    dev.hold()
    nr_fill(nr, 'grey80')                    # Clear the nativeRaster
    nr_blit(nr, dino[[(i %% 16) + 1]], i, 1) # copy dino to nativeRaster
    grid.raster(nr, interpolate = FALSE)     # copy nativeRaster to screen
    Sys.sleep(0.02)   # Stop animation running too fast.
    dev.flush()                          
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.