Code Monkey home page Code Monkey logo

ropensci / googlelanguager Goto Github PK

View Code? Open in Web Editor NEW
190.0 22.0 42.0 8.86 MB

R client for the Google Translation API, Google Cloud Natural Language API and Google Cloud Speech API

Home Page: https://code.markedmondson.me/googleLanguageR/

License: Other

R 35.55% JavaScript 7.52% CSS 0.56% Dockerfile 0.07% HTML 56.30%
cloud-speech-api cloud-translation-api translation-api googleauthr google-api-client r rstats google-cloud-speech google-cloud sentiment-analysis google-nlp natural-language-processing speech-api r-package peer-reviewed

googlelanguager's Introduction

googleLanguageR - R client for the Google Translation API, Natural Language API, Speech-to-Text API and Text-to-Speech API

CRAN Build Status codecov.io

Language tools for R via Google Machine Learning APIs

Read the introduction blogpost on rOpenSci's blog

This package contains functions for analysing language through the Google Cloud Machine Learning APIs

Note all are paid services, you will need to provide your credit card details for your own Google Project to use them.

The package can be used by any user who is looking to take advantage of Google’s massive dataset to train these machine learning models. Some applications include:

  • Translation of speech into another language text, via speech-to-text then translation and having the results spoen back to you
  • Talking Shiny apps
  • Identification of sentiment within text, such as from Twitter feeds
  • Pulling out the objects of a sentence, to help classify texts and get metadata links from Wikipedia about them.

The applications of the API results could be relevant to business or researchers looking to scale text analysis.

Google Natural Language API

Google Natural Language API reveals the structure and meaning of text by offering powerful machine learning models in an easy to use REST API. You can use it to extract information about people, places, events and much more, mentioned in text documents, news articles or blog posts. You can also use it to understand sentiment about your product on social media or parse intent from customer conversations happening in a call center or a messaging app.

Read more on the Google Natural Language API

Google Cloud Translation API

Google Cloud Translation API provides a simple programmatic interface for translating an arbitrary string into any supported language. Translation API is highly responsive, so websites and applications can integrate with Translation API for fast, dynamic translation of source text from the source language to a target language (e.g. French to English).

Read more on the Google Cloud Translation Website

Google Cloud Speech-to-Text API

Google Cloud Speech-to-Text API enables you to convert audio to text by applying neural network models in an easy to use API. The API recognizes over 80 languages and variants, to support your global user base. You can transcribe the text of users dictating to an application’s microphone or enable command-and-control through voice among many other use cases.

Read more on the Google Cloud Speech Website

Google Cloud Text-to-Speech API

Google Cloud Text-to-Speech enables developers to synthesize natural-sounding speech with 30 voices, available in multiple languages and variants. It applies DeepMind’s groundbreaking research in WaveNet and Google’s powerful neural networks to deliver the highest fidelity possible. With this easy-to-use API, you can create lifelike interactions with your users, across many applications and devices.

Read more on the Google Cloud Text-to-Speech Website

Installation

  1. Create a Google API Console Project
  2. Within your project, add a payment method to the project
  3. Within your project, check the relevant APIs are activated
  1. Generate a service account credential as a JSON file by first creating a service account and then creating credentials for a service account
  2. Return to R, and install the official release via install.packages("googleLanguageR"), or the development version with remotes::install_github("ropensci/googleLanguageR")

Docker image

Some Docker images are publicly available. In general gcr.io/gcer-public/googleLanguageR:$BRANCH_NAME carries that GitHub branch's version.

  • gcr.io/gcer-public/googleLanguageR:CRAN - the latest CRAN version CRAN
  • gcr.io/gcer-public/googleLanguageR:master - latest GitHub master version Build Status
  • gcr.io/gcer-public/googleLanguageR:feature - a feature branch from GitHub

Usage

Authentication

The best way to authenticate is to use an environment file. See ?Startup. I usually place this in my home directory. (e.g. if using RStudio, click on Home in the file explorer, create a new TEXT file and call it .Renviron)

Set the file location of your download Google Project JSON file in a GL_AUTH argument:

#.Renviron
GL_AUTH=location_of_json_file.json

Then, when you load the library you should auto-authenticate:

library(googleLanguageR)

You can also authenticate directly using the gl_auth function pointing at your JSON auth file:

library(googleLanguageR)
gl_auth("location_of_json_file.json")

You can then call the APIs via the functions:

  • gl_nlp() - Natural Langage API
  • gl_speech() - Cloud Speech-to-Text API
  • gl_translate() - Cloud Translation API
  • gl_talk() - Cloud Text-to-Speech API

Natural Language API

The Natural Language API returns natural language understanding technolgies. You can call them individually, or the default is to return them all. The available returns are:

  • Entity analysis - Finds named entities (currently proper names and common nouns) in the text along with entity types, salience, mentions for each entity, and other properties. If possible, will also return metadata about that entity such as a Wikipedia URL. If using the v1beta2 endpoint this also includes sentiment for each entity.
  • Syntax - Analyzes the syntax of the text and provides sentence boundaries and tokenization along with part of speech tags, dependency trees, and other properties.
  • Sentiment - The overall sentiment of the text, represented by a magnitude [0, +inf] and score between -1.0 (negative sentiment) and 1.0 (positive sentiment).

Demo for Entity Analysis

You can pass a vector of text which will call the API for each element. The return is a list of responses, each response being a list of tibbles holding the different types of analysis.

texts <- c("to administer medicince to animals is frequently a very difficult matter, and yet sometimes it's necessary to do so", 
         "I don't know how to make a text demo that is sensible")
nlp_result <- gl_nlp(texts)

# two results of lists of tibbles
str(nlp_result, max.level = 2)

See more examples and details on the website or via vignette("nlp", package = "googleLanguageR")

Google Translation API

You can detect the language via gl_translate_detect, or translate and detect language via gl_translate

Note this is a lot more refined than the free version on Google’s translation website.

text <- "to administer medicine to animals is frequently a very difficult matter, and yet sometimes it's necessary to do so"
## translate British into Danish
gl_translate(text, target = "da")$translatedText

See more examples and details on the website or via vignette("translate", package = "googleLanguageR")

Google Cloud Speech-to-Text API

The Cloud Speech-to-Text API provides audio transcription. Its accessible via the gl_speech function.

A test audio file is installed with the package which reads:

“To administer medicine to animals is frequently a very difficult matter, and yet sometimes it’s necessary to do so”

The file is sourced from the University of Southampton’s speech detection (http://www-mobile.ecs.soton.ac.uk/newcomms/) group and is fairly difficult for computers to parse, as we see below:

## get the sample source file
test_audio <- system.file("woman1_wb.wav", package = "googleLanguageR")

## its not perfect but...:)
gl_speech(test_audio)$transcript


    ## # A tibble: 1 x 2
    ##   transcript                                                    confidence
    ##   <chr>                                                         <chr>     
    ## 1 to administer medicine to animals is frequency of very diffi… 0.9180294

See more examples and details on the website or via vignette("speech", package = "googleLanguageR")

Google Cloud Text-to-Speech API

The Cloud Text-to-Speech API turns text into talk audio files. Its accessible via the gl_talk function.

To use, supply your text to the function:

gl_talk("This is a talking computer.  Hello Dave.")

See more examples and details on the website or via vignette("text-to-speech", package = "googleLanguageR")

ropensci_footer

googlelanguager's People

Contributors

davedgd avatar dietrichson avatar howardbaek avatar maelle avatar markedmondson1234 avatar muschellij2 avatar retowyss avatar smmurphy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

googlelanguager's Issues

Error: API returned: sample_rate_hertz (16000)

Hi Mark,

I was trying to run some basic voice memo transcribing through the package. But I keep getting this error.

test_audio <- "gen.wav"
gl_speech(test_audio)$transcript

Error: API returned: sample_rate_hertz (16000) in RecognitionConfig must either be omitted or match the value in the WAV header ( 48000).

What's the best way to set the parameter to overcome this?

rOpenSci?

Maybe it overlaps other packages a bit, but would love the peer review
https://github.com/ropensci/onboarding

Have to comply with the package name must be lower case thing though. I disagree with this as it makes some packages completely un-Googleable, but I guess their promotion on openSci websites make up for this.

API limit reached, need catch

After running a bunch of requests on gl_translate_detect, I ran in to this problem:

Request failed [403]. Retrying in 1 seconds...
Request failed [403]. Retrying in 1 seconds...
2018-11-20 11:17:24> Request Status Code: 403
Scopes: https://www.googleapis.com/auth/cloud-language https://www.googleapis.com/auth/cloud-platform
Method: service_json
Error in if (length(string) == nrow(me)) { : argument is of length zero

Description of the problem: After running into a failure, the Reduce argument will return NULL to me, so (length(string) == nrow(me)) will fail. I think a catch in there for NULL could help.

I've tried to set the option googleAuthR.HttrRetryTimes higher as located in https://github.com/MarkEdmondson1234/googleAuthR/blob/91a405930ee53a27ee02a5fe3d87be3f3f9fc617/R/options.R, but that doesn't seem to make a difference. Any ideas?

false character count when using Umlauts

Hi all

I have not used your package so far, I wrote my one script to call the REST API with rjson. However, I have noticed a problem when using Google's analyzeSyntax and analyzeEntities, and I was wondering if you have the same issue:

I am analyzing a german text with Umlauts (ä,ö,ü,..) and Google seems to count all Umlauts as two characters, which means that all values for the offsets are wrong after the first appearance of an Umlaut in the text. So basically I can't map the tokens or the entities back to the original text. Have you had this issue (maybe with other special characters) and how do you deal with it? I am encoding with UTF8, so it should not be an encoding problem. I am working with a windows environment.

Thanks and best wishes from Oslo
Fabienne

Implement Speaker diarization

Hi,

Thank you for creating a nifty bit of code with this package. It appears that Google is now offering a beta version of Speaker diarization (ability to identify speaker's voice). I am wondering if this could be implemented in the googleLanguageR package.

Thanks,

Eric

Update purrr requirement to > 0.2.4

Hi,
first of all, thank you for the package!

I am getting the below error message when running the code:

library(googleLanguageR)
gl_auth("<myJson>.json")
nlp_result <- gl_nlp("Hello World")

2018-03-13 23:45:11 -- annotateText: 5 characters
Error: 'as_mapper' is not an exported object from 'namespace:purrr'

I tried reinstalling the "purrr" package but it gives the same error message. I got the package from Github ("ropensci/googleLanguageR). I am running R 3.3.3

Any ideas?

Thank you,
Jan

Sort the bleeding tests out

You may also want to consider using the httptest package to help you have cleaner API fixtures that don't include your auth token. httptest can also help you to get to 100% line coverage easily: you can assert that the right requests are made with the various API versions and "gs://" URLs (things that are among your untested lines currently) without needing to actually make all of those requests or maintain mocks for their responses. See the package vignette for some examples.
Even where you do have line coverage, the tests are thin, particularly in showing what the API response object looks like. This is particularly important because, as I mentioned above, the man pages also don't give much guidance on what the API responses look like. As a user, I'd like to have some kind of reference for what to expect that the functions return, and with respect to tests, some kind of assurance of the data contract. While the man pages ideally should describe the responses in more detail, you can also use tests to give more detailed examples that show the shape of a return object, possibly more naturally than you can in the @examples section of the docs.

Error: API returned: Daily Limit Exceeded - is there a temp file?

I tried to use the package to translate some text. I checked the nchar amount, which is within the daily limit (https://cloud.google.com/translate/quotas)

sum(nchar(articles_to_translate))
[1] 1999332

I then used the command

articles_translated <- gl_translate(articles_to_translate, 
                                    target = "en",
                                    format = "text",
                                    source = "it")

which seemed to work until I received a daily limit exceeded error. Now Google already billed me, but no results were returned in R.
Is there a way to still get the result for example from temporary files somewhere on my harddisk?

Thanks for your help, sessioninfo below:

> sessionInfo()
R version 3.5.0 (2018-04-23)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

Matrix products: default

locale:
[1] LC_COLLATE=German_Germany.1252  LC_CTYPE=German_Germany.1252    LC_MONETARY=German_Germany.1252
[4] LC_NUMERIC=C                    LC_TIME=German_Germany.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] googleLanguageR_0.2.0 rDNA_2.1.6            dplyr_0.7.5           rJava_0.9-9     

gl_speech output is incomplete

Hi Mark,

Having a bit of a strange issue -- I can get gl_speech to run with the code below, but it seems to cut the transcript short.:

library(googleLanguageR)
library(tuneR)

a <- readWave("OSR_us_000_0018_8k.wav", from = 0, to = 1, units = "minutes")
b <- mono(a)
writeWave(b, "OSR_us_000_0018_8k_new.wav", extensible = FALSE)
text <- gl_speech("OSR_us_000_0018_8k_new.wav", sampleRateHertz = [email protected])

The resulting transcript is correct, but only represents the first ~6 seconds of the new wav file. I've listened to the new file, and it contains speech for at least 30 seconds. I can replicate this issue with a different wav file. Any insight? Thanks!

Error while using gl_speech()

Hi,

I am trying to convert the wav file to text, i.e. audio to text. Where i am using googleLanguageR package to convert the data, for this i am using gl_speech(test_audio) method, its throwing an error saying "2018-01-17 20:22:02> No authorization yet in this session!
2018-01-17 20:22:02> No .httr-oauth file exists in current working directory. Do library authentication steps to provide credentials.
Error: Invalid token

Please can anyone help me in debugging this issue.

Parsing Error with audio files longer than 60 sec

Hi all,
I'm using the googleLanguageR package version 0.2.0.9 to transcribe German phone calls to text with the Google Speech-to-text API (speaker diarization is turned on, two speakers).
However, whenever I want to transcribe a file, which is longer than 60 seconds (i.e., I store it in a Cloud Bucket and then access it via the URI) it gives me a warning message.

Here is my code:

my_config <- list(encoding = "LINEAR16",
                  enableSpeakerDiarization = TRUE,
                  diarizationSpeakerCount = 2)

testcall <- "gs://[bucket]/testcall.wav"

apicall<- gl_speech(testcall, sampleRateHertz = 8000, languageCode = "de-DE", asynch = TRUE, customConfig = my_config)

testcall_transcript <- gl_speech_op(apicall)

The transcription is successful but R gives me this warning message.

Warning message:
In value[[3L]](cond) : Could not parse object with names: 

What this error causes is that the structure of the two returend dataframes seems to be a little mixed up.
When I call str(testcall_transcript) it gives me the following output:

List of 2
 $ transcript:Classes ‘tbl_df’, ‘tbl’ and 'data.frame':	2 obs. of  2 variables:
  ..$ transcript: chr [1:2] "ja hallo ja und vergebe Zusatzdaten und zwar hat er was mache ich als nicht anzumerken ist einfach machen 815" "und wie heißt die Variable die drin da diese Datei ein Kratzer nennst Zusatzdaten Zusatzdaten vorgangs-id nicht"| __truncated__
  ..$ confidence: chr [1:2] "0.8421874" "0.8393924"
 $ timings   :List of 2
  ..$ :'data.frame':	1 obs. of  3 variables:
  .. ..$ transcript: chr "ja hallo ja und vergebe Zusatzdaten und zwar hat er was mache ich als nicht anzumerken ist einfach machen 815"
  .. ..$ confidence: num 0.842
  .. ..$ words     :List of 1
  .. .. ..$ :'data.frame':	20 obs. of  3 variables:
  .. .. .. ..$ startTime: chr [1:20] "0s" "17.600s" "18.100s" "18.200s" ...
  .. .. .. ..$ endTime  : chr [1:20] "17.600s" "18.100s" "18.200s" "19.100s" ...
  .. .. .. ..$ word     : chr [1:20] "ja" "hallo" "ja" "und" ...
  ..$ :'data.frame':	1 obs. of  3 variables:
  .. ..$ transcript: chr "und wie heißt die Variable die drin da diese Datei ein Kratzer nennst Zusatzdaten Zusatzdaten vorgangs-id nicht"| __truncated__
  .. ..$ confidence: num 0.839
  .. ..$ words     :List of 1
  .. .. ..$ :'data.frame':	43 obs. of  4 variables:
  .. .. .. ..$ startTime : chr [1:43] "0s" "17.600s" "18.100s" "18.200s" ...
  .. .. .. ..$ endTime   : chr [1:43] "17.600s" "18.100s" "18.200s" "19.100s" ...
  .. .. .. ..$ word      : chr [1:43] "ja" "hallo" "ja" "und" ...
  .. .. .. ..$ speakerTag: int [1:43] 1 1 1 1 1 1 1 1 1 1 ...

Looks all fine BUT...
when I try to access the $timings dataframe I'm having trouble to access the $speakerTag variable. I need to access the speakerTag and the respective start and Endtimes in order to determine the time stamps when a speaker turn happens.

For a short file (less than 60sec) R gives me this output (perfectly working):

> transcript_short$timings$speakerTag
  [1] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 [68] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1

For the long file R gives me this output:

> testcall_transcript$timings$speakerTag
NULL

Any ideas on how this can be fixed? Extracting the speakerTags is crucial for my further data processing.
Thanks! :)

Issue with error code 403

@Any suggestions on how to solve this issue. I am getting the following error in R:

Listening on http://127.0.0.1:7522
Calling Speech API
Request failed [403]. Retrying in 1.7 seconds...
Request failed [403]. Retrying in 3.7 seconds...
2018-03-05 22:27:42> Request Status Code: 403
Scopes: https://www.googleapis.com/auth/cloud-platform
Method: service_json
Warning: Error in : API returned: Google Cloud Speech API has not been used in project cool-bay-197110 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/speech.googleapis.com/overview?project=cool-bay-197110 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry. @MarkEdmondson1234

Check if parse_speech and parse_asynch are necessary now

Note: parse_speech and parse_async are now quite similar. I'm not sure of the original intention of the separation and whether both are truly needed. Prior to the last pull request it seems that parse_speech was being called for asynchronous transcriptions and parse_async was unused.

via #32

Error returned for gl_speech_op()

First, thank you so much for your work on this package! I have been using this very successfully on roughly 5 minute files with the following:

op <- gl_speech("file",
          encoding = "FLAC",
          sampleRateHertz = 48000L,
          speechContexts = list(phrases = hints)$transcript,
          asynch = TRUE)
text <- gl_speech_op(op)

I just tried with a longer file, 45 minutes, using the same code as above and received the following:

Error in rbind(deparse.level, ...) : 
  numbers of columns of arguments do not match
Warning message:
In call_api() :
  API Data failed to parse.  Returning parsed from JSON content.
                    Use this to test against your data_parse_function.

I verified that the above code still works perfectly well on the 5 minute files I was working with previously. Any idea what might be going on?

Error: Path '' does not exist

Firstly, thanks! You've done a great service to the R community by providing this package. But sadly, I can't get it to work. After restarting the R session in RStudio:

`
Welcome at Tue Sep 25 11:43:39 2018

Sys.setenv(GL_AUTH = "/home/john/Dropbox/Writing/Software/Go/src/snp/google-service-account.json")
googleLanguageR::gl_auth("/Dropbox/Writing/Software/Go/src/snp/google-service-account.json")
library(googleLanguageR)
test_audio <- system.file("
/Documents/test-for-google.flac", package = "googleLanguageR")
result <- gl_speech(test_audio, languageCode = "en-GB")
Error: Path '' does not exist`

Maybe this will help?

`

sessionInfo()
R version 3.5.0 (2018-04-23)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Fedora 28 (Workstation Edition)

Matrix products: default
BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so

locale:
[1] LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=C LC_TIME=en_GB.UTF-8 LC_COLLATE=en_GB.UTF-8
[5] LC_MONETARY=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8 LC_PAPER=en_GB.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] googleLanguageR_0.2.0

loaded via a namespace (and not attached):
[1] digest_0.6.17 crayon_1.3.4 assertthat_0.2.0 R6_2.2.2 jsonlite_1.5 magrittr_1.5
[7] pillar_1.3.0 httr_1.3.1 rlang_0.2.2 curl_3.2 rstudioapi_0.7 googleAuthR_0.6.3
[13] tools_3.5.0 purrr_0.2.5 yaml_2.2.0 compiler_3.5.0 base64enc_0.1-3 memoise_1.1.0
[19] openssl_1.0.2 tibble_1.4.2
`

Async call does not return the entire transcript

Taking a wild shot here hoping you have seen this. I am trying to transcribe a 2+ minute audio file asynchronously. I am able to send the audio file and return the results. However, I am only getting 18 seconds back. Checking the google API dashboard, I see the entire 2 minute file is processed. I have checked with google support and all looks well there.

Have you seen any issues with not all of the file being returned. I am running R 3.4.1 and googleLanguageR 0.1.0.9000 on a MAC 10.12.6

Testwav3 <- ("gs://qual_audios/audio_only3.flac")
Testasync3 <- gl_speech(Testwav3, encoding = "FLAC", sampleRateHertz = 16000L, asynch = TRUE)
Test_result3 <- gl_speech_op(Testasync3)

Error: API returned: Must use single channel (mono) audio, but WAV header indicates 2 channels.

Hi all,

I use the following code:

gl_speech("test.wav")

I get:

"Request failed [400]. Retrying in 1 seconds...
Request failed [400]. Retrying in 2.5 seconds...
2018-03-05 11:57:40> Request Status Code: 400
Scopes: https://www.googleapis.com/auth/cloud-platform
App key: 564266350373-q5ggqe4k894orbfbu5nfq85g1dafvnl9.apps.googleusercontent.com
Method: new_token
Error: API returned: sample_rate_hertz (48000) in RecognitionConfig must either be omitted or match the value in the WAV header ( 44100)."

any ideas?

many thanks
David

and when supplying sample rate:

gl_speech("test.wav", sampleRateHertz=44100L)
Request failed [400]. Retrying in 1.8 seconds...
Request failed [400]. Retrying in 1.3 seconds...
2018-03-05 15:31:38> Request Status Code: 400
Scopes: https://www.googleapis.com/auth/cloud-platform
App key: 564266350373-q5ggqe4k894orbfbu5nfq85g1dafvnl9.apps.googleusercontent.com
Method: new_token
Error: API returned: Must use single channel (mono) audio, but WAV header indicates 2 channels.

and then..

what is Wave() function?
I get Error in (function (classes, fdef, mtable) :
unable to find an inherited method for function ‘Wave’ for signature ‘"Wave"’

Graceful handling of errors in vectorised inputs

I am trying to loop through a dataframe with reviews. Some of which seem to be below the detection threshold. I get errors like

When I try to do something like this:

for (i in 1:nrow(df_filtered[1:10,])) {
      tmp <- safely(gl_nlp(df_filtered$review_text[i]))
      
      api_result[[as.character(df_filtered$id[i])]] <- tmp
      
      print(paste0("Index: ",i," Status: ",length(tmp)))
 }

I get errors like:

2019-02-27 14:50:58 -- annotateText: 65 characters
Request failed [400]. Retrying in 1 seconds...
Request failed [400]. Retrying in 1 seconds...
2019-02-27 14:51:03> Request Status Code: 400
Scopes: https://www.googleapis.com/auth/cloud-language https://www.googleapis.com/auth/cloud-platform
Method: service_json
Error: API returned: Invalid text content: too few tokens (words) to process.

I was thinking why should the whole loop error out due to only 1 bad call. I am using the safely function from purrr. But is there a best practise guide for dealing with these situations somewhere?

Thanks.

Sort out the non-ASCII text that will block CRAN

You have non-ASCII text in test files, which if I recall correctly from a previous CRAN rejection of mine, will cause problems when they run your tests on Solaris. You can resolve this by moving the non-ASCII text to a separate file that gets sourced or otherwise read in and place that after a skip_on_cran. Just placing it in a test-file after a skip_on_cran isn't sufficient because the test file will fail to source. Cf. Crunch-io/rcrunch@7f74971

Typo on your README

Hi guys, not a real user, but I just bumped to this repo.

Two of your headlines, 1 and 2 read "Google Cloud Speech-to-Text API", when I think you meant "Google Cloud Text-to-Speech API".

JSON fetch error: Invalid JSON payload received. Unknown name "extract_entity_sentiment" at 'features': Cannot find field.

R 3.4
Installed googleLanguageR via devtools today.

Running the basic sample query to Natural Language API from your code results in error.

text <- "to administer medicince to animals is frequently a very difficult matter, and yet sometimes it's necessary to do so"
nlp_result <- gl_nlp(text)

Error:
2017-05-04 12:25:40 -- annotateTextanalyzeEntitiesanalyzeSentimentanalyzeSyntaxanalyzeEntitySentiment for 'to administer medicince to animals is frequently a very difficult matter, and yet sometimes it's nec...' Request Status Code: 400 Error in checkGoogleAPIError(req) : JSON fetch error: Invalid JSON payload received. Unknown name "extract_entity_sentiment" at 'features': Cannot find field.

Thoughts?

Improve gl_language_detect I/O

I gave it the same column of text data as before.

lang <- gl_language_detect(df$comments)

which errored with "Total URL must be less than 2000 characters"

  • What should one do in this case? A more constructive error message would help.
  • Why? Does the API restrict this way? I find that surprising, particularly since you're doing a POST request, which in principle could take a very large request body.

I selected a subset of the column and retried:

> lang <- gl_translate_detect(df$comments[995:997])
Detecting language: 1879 characters -                                                   SOME QUESTIONS DIDNT HAVE GOOD CHOICES. THE RIGHT                                                   ...
> lang
[[1]]
  confidence isReliable language
1          1      FALSE      und

[[2]]
  isReliable language confidence
1      FALSE       en  0.9981712

[[3]]
  isReliable language confidence
1      FALSE      und          1

Again, would really help if the man page documented the return object in more detail.

Seems like this would more logically return as a single data.frame rather than a list of 1-row data.frames. Much more easily analyzed and manipulated.

Document uploaded to Cloud Storage when: 'Inline audio exceeds duration limit' on `asynch = T` calls

I had the following error when trying to do an asynch call. @Bellman281 reported this issue here. Did the google API change?

asynch <- gl_speech("data/TEMP/CPC.wav", asynch = T, languageCode = "es-ES")

Request failed [400]. Retrying in 1 seconds...
Request failed [400]. Retrying in 1.9 seconds...
2018-06-12 00:00:44> Request Status Code: 400
Scopes: https://www.googleapis.com/auth/cloud-platform
Method: service_json
Error: API returned: Inline audio exceeds duration limit. Please use a GCS URI.

How to get Authentication?

Authentication
The best way to authenticate is to use an environment file. See ?Startup. I usually place this in my home directory. (e.g. if using RStudio, click on Home in the file explorer, create a new TEXT file and call it .Renviron)
Set the file location of your download Google Project JSON file in a GL_AUTH argument:

GL_AUTH=location_of_json_file.json

I try to browse over http://code.markedmondson.me/googleLanguageR/articles/speech.html but don;t know how to get authentication in order to use the functions.

> library('googleLanguageR')
> lnk <- 'https://scontent-yyz1-1.xx.fbcdn.net/v/t66.18014-6/42832823_1854945517918295_8803608342512894293_n.mp4?_nc_cat=110&efg=eyJ2ZW5jb2RlX3RhZyI6Im9lcF9oZCJ9&oh=08e0a1bce580184048cd33489d3c254c&oe=5C3DAE2F'
> vfile <- tempfile()
> 
> if (file.exists(vfile)) {
+   vday <- gl_speech(vfile)
+   
+   play(vday)
+ } else {
+   download.file(lnk, vfile, mode = 'wb')
+   vday <- gl_speech(vfile)
+   #play(vday)
+ }
trying URL 'https://scontent-yyz1-1.xx.fbcdn.net/v/t66.18014-6/42832823_1854945517918295_8803608342512894293_n.mp4?_nc_cat=110&efg=eyJ2ZW5jb2RlX3RhZyI6Im9lcF9oZCJ9&oh=08e0a1bce580184048cd33489d3c254c&oe=5C3DAE2F'
Content type 'video/mp4' length 5844502 bytes (5.6 MB)
downloaded 5.6 MB

2018-10-18 22:42:25> No authorization yet in this session!
2018-10-18 22:42:25> NOTE: a  .httr-oauth  file exists in current working directory.
 Run authentication function to use the credentials cached for this session.
Error: Invalid token

Issue with error code 404

Hello. I'm having difficulty transcribing audio into text. R is returning the following error:

2019-01-22 23:00:10 -- Using Google Storage URI: gs://storage.cloud.google.com/entrevista181113_003/teste.flac
Request failed [404]. Retrying in 1 seconds...
Request failed [404]. Retrying in 1.7 seconds...
2019-01-22 23:00:15> Request Status Code: 404
Scopes: https://www.googleapis.com/auth/cloud-language https://www.googleapis.com/auth/cloud-platform
Method: service_json
Error: API returned: Requested entity was not found.

Can someone help me? Thank you.

gl_nlp() pukes w/ parameter nlp_type="analyzeSyntax"

The code below pukes if nlp_type is specified. If it isn't, it works fine, but I am trying to restrict the nlp performed to get speedier results.

library(googleLanguageR)
gl_auth("<blablabla.json>")
gl_nlp("Two Things are Infinite",
nlp_type="analyzeSyntax" # if i omit this it works
)

I get the following error:

2018-02-16 17:21:50 -- analyzeSyntax: 23 characters
Error in data.frame(..., check.names = FALSE) :
arguments imply differing number of rows: 1, 0
Error: Result 1 is not a length 1 atomic vector
In addition: Warning message:
In call_api(the_body = body) :
API Data failed to parse. Returning parsed from JSON content.
Use this to test against your data_parse_function.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.