globaltrends 0.2.1

Data Structure

• Change from sqlite/parquet files to data.table for performance reasons
  • All globaltrends data is stored in a single, non-compressed RDS file
  • The data is stored in a list objectiv with the following data.table objects:
    • batch_keywords
    • batch_time
    • data_control
    • data_object
    • data_score
    • data_doi
    • data_locations
    • data_region
    • data_related
    • keyword_synonyms
  • All globaltrends data is loaded into memory as a list of data.table objects
  • This makes the storage structure incompatiable with the structure used up to 0.2.0
    • It is possible to integrate the data manually. To this end, extract the data from the parquet files, transform it to data.table objects and enter it into the respective elements in the gt.env environment.
• Drop vacuum_data() function, since the data architecture is not built on a database file any more

Efficiency

• compute_score() now persists in-memory data to disk after every 1,000 computed locations, matching the periodic persistence already used by the download_xxx() functions

Testing

• All tests now run fully isolated, in their own temporary directory, with automatic cleanup (withr) - no shared state or leftover files between tests
• Live Google Trends API calls in the test suite are now opt-in (set GLOBALTRENDS_LIVE_TESTS=1), so automated test runs are offline and deterministic by default
• Added test coverage for previously untested internals: API retry/backoff logic, quota handling, input validators, database persistence, and initialize_python()
• Enabled parallel test execution (Config/testthat/parallel: true)

globaltrends 0.2.0

• General revision and improvement of code base
• General revision and improvement of tests
• General revision and improvement of documentation
• Inclusion of API-usage counter for downloads through Google Trends API (10,000 daily downloads)

Change in database handling

• Switch from sqlite to parquet
  • Database tables are stored as individual parquet files
  • start_db() loads the parquet files to memory
  • All database actions are conducted in memory and later written to parquet
  • The download functions write in-memory data to parquet after every 1,000 downloads

Technical issues and bug fixes

• Updated dependency to testthat 3.0.0
• Improved error handling for Google Trends API calls: .get_trend() now catches three classes of errors from the Research API backend (HTTP via reticulate) instead of crashing with a Python traceback
  • Quota exceeded (HTTP 429): writes in-memory data to parquet and then aborts with a clear message asking the user to wait for the quota to reset.
  • Invalid argument (HTTP 400): skips the offending term/location/date combination silently and continues.
  • Timeout (WinError 10060): skips the affected location and continues; the missing data can be filled by re-running download_object().

globaltrends 0.1.0

Change in downloading approach

• Run downloads from Google Trends through the official Google Trends API
  • Access available from Google: https://support.google.com/trends/contact/trends_api
  • Downloads are made with the Google API Python Client
  • globaltrendsuses reticulate to run the respective Python script
• Users can choose to download through the official API or the gtrendsrpackage
  • If users call initialized_python(), the downloads will be made through the official Google Trends API
  • If users run downloads without calling initialized_python(), the downloads will be made through the gtrendsrpackage
• Control the waiting time between queries through the environmental variable gt.env$query_wait (default value = 0.1s)

Package simplifications

• Remove time-series adjustments from compute_score for package simplification
• Remove get_abnorm_hist function for package simplification
• Remove plotting functions and export classes for package simplification
• Accelerate compute_score
• Accelerate handling of synonyms in separate function aggregate_synonyms

Technical issues and bug fixes

• Increase dependencies to dplyr 1.1.1 and replace the multiple argument by relationship
• Fixed issues in sample data

globaltrends 0.0.14

• Remove dependency on glue package, calls of glue::glue are replaced with paste and paste0
• Remove dependency on ineq package, calls of ineq::ineq are replaced with internal functions
• Use functions from DBI 1.1.0: dbCreateTable and dbAppendTable
• Remove the function plot_map from the package:
  • The function proofed too inflexible, only world map was included
  • We point users to the functions from the packages maps and ggmap
  • Remove dependency on the maps package
• Remove reference to non-existent table data_global in function remove_data

globaltrends 0.0.13

• Set minimum dependency of stats and utils to version 3.5.0 in line with minimum R dependency
• Store globaltrends logo in the vignettes folder
• Add explanation of Google’s data preparation methodology to vignette
• Change wait intervals for status responses != 200 and add message about automatic retry of download
  • Status == 500: wait 1 second
  • All other responses: wait 60 seconds
• Add function vacuum_data to free unused memory after remove_data
• Remove all usage of .data to comply with tidyselect 1.2.0, this applies to calls of:
  • dplyr::rename
  • dplyr::select
  • purrr::map
  • purrr::walk
  • tidyr::nest
  • tidyr::pivot_longer
  • tidyr::pivot_wider
  • tidyr::unnest
• As a consequence of the changes in tidyselect 1.2.0, several objects are defined as global variables (see globals.r for details)
• Replace size with linewidth to comply with ggplot2 3.4.0
• Add the argument multiple = "all" and multiple = "error" to comply with dplyr 1.1.0
• Facilitate handling of location = “NA - Namibia”, the function add_locations does automatically drop the location
• Remove dependency on WDI package, references to WDI::WDI_data are replaced by data objects countries and countries_wdi

globaltrends 0.0.12

• Stop direct exports from functions to .GlobalEnv
• Exports of objects (e.g., SQL connections, keyword tables) are redirected to package environment gt.env
• Add handle for Namibia ISO code

globaltrends 0.0.11

• Add function plot_map()
• Add automatic stringr::str_squish for keywords and synonyms
• Add documentation on usage of search topics

globaltrends 0.0.10

• Change indication and handling of synonyms (synonyms are now ignored in export_score and export_voi)

globaltrends 0.0.9

• Fasten computation of search scores for data with synonyms
• Fix bugs for control keywords that have a mean of 0 for their search volume time series

globaltrends 0.0.8

• Adapt export_xxx for vector and list inputs
• Optimize RAM usage of export_xxx
• Check whether “db/globaltrends_db.sqlite” file exists in working directory
• Inclusion of “workaround” plot functions that set class xxx automatically:
  • plot_xxx_box
  • plot_xxx_bar
  • plot_xxx_ts

globaltrends 0.0.7

• Change waiting times for errors
  • Status Code != 200/Limit exceeded -> 60 seconds wait
  • Status Code == 500 -> 1 second wait

globaltrends 0.0.6

• To distinguish them from the actual database tables, names of the example data object were adapted. Documentation is still available for the respective database table.
  • batch_keywords -> example_keywords
  • batch_time -> example_time
  • data_control -> example_control
  • data_doi -> example_doi
  • data_object -> example_object
  • data_score -> example_score
• Waiting period between downloads reduced from 20-30 seconds to 5-10 seconds

globaltrends 0.0.5

• Added classes for output from export_xxx
  • export_score -> class(“exp_score”)
  • export_voi -> class(“exp_voi”)
  • export_doi -> class(“exp_doi”)
• export_xxx + get_abnorm_hist supersedes export_xxx_change
  • method for export_score -> class(“exp_score”)
  • method for export_voi -> class(“exp_voi”)
  • method for export_doi -> class(“exp_doi”)
• plot_bar supersedes plot_score
  • method for export_score -> class(“exp_score”)
  • method for export_score -> get_abnorm_hist -> class(“abnorm_score”)
• plot_ts supersedes plot_voi_ts and plot_doi_ts
  • method for export_score -> class(“exp_score”)
  • method for export_score -> get_abnorm_hist -> class(“abnorm_score”)
  • method for export_voi -> class(“exp_voi”)
  • method for export_voi -> get_abnorm_hist -> class(“abnorm_voi”)
  • method for export_doi -> class(“exp_doi”)
  • method for export_doi -> get_abnorm_hist -> class(“abnorm_doi”)
• plot_box supersedes plot_voi_box and plot_doi_box
  • method for export_score -> class(“exp_score”)
  • method for export_score -> get_abnorm_hist -> class(“abnorm_score”)
  • method for export_voi -> class(“exp_voi”)
  • method for export_voi -> get_abnorm_hist -> class(“abnorm_voi”)
  • method for export_doi -> class(“exp_doi”)
  • method for export_doi -> get_abnorm_hist -> class(“abnorm_doi”)