The Slog request type report run against https://oik-plugins.uk/bwtrace/bwtrace.vt.20210313 reported 71 GET-404's
but viewing the file I can only see one. What are the other 70?

Additionally, GET-404 isn't a request type I can filter on.

As an aside. I also see requests from IP address 127.0.0.1 taking varying amounts of time.

/,,3.981695,7.3.27,2109,5304,1550,34,1813,55,36,16,66,,,,,78300,127.0.0.1,3.416422,2021-03-13T20:10:53+00:00,,GET,
/,,1.051810,7.3.27,2109,5313,1550,34,1813,55,36,16,64,,,,,79306,127.0.0.1,1.050910,2021-03-13T20:10:55+00:00,,GET,
/,,14.107096,7.3.27,2109,5304,1555,34,1818,55,36,16,75,,,,,80425,127.0.0.1,14.034743,2021-03-13T20:11:09+00:00,,GET,

These don't report an HTTP response. Why not?

While downloading files from wp-a2z.org I notice that files greater than a certain size were not being downloaded. They were being saved as 0 byte files.

This problem needs to be investigated.

When a record in the daily trace summary file is badly formed
eg a record that only contains 200
then a Warning message is issued for the Elapsed report type.

Warning: A non-numeric value encountered in C:\apache\htdocs\wordpress\wp-content\plugins\slog\libs\class-vt-stats.php on line 461

This is due to the field's value ($this->field_value) being set to " " in Object_Grouper::group.
Setting the value to null resolves this issue.

The current filtering options which are run on some tabs can filter requests by request type ( eg only return GET-FE - Front end GET requests ). This will still list any URL requested. When running comparisons of daily trace summary files it would be nice to compare daily trace summary files for a single URL, or range of URLs without having to use the Driver functionality.

Requirement

• Add a Filter URL input field to be used to filter requests when the URL matches.

Proposed solution

• Add the Filter URL field on the Settings tab ( _slog_filter_url )
• Implement the filter URL logic
• Implement as overrides for each of the relevant tabs:
• Reports - YES
• Compare - YES
• Filter _ Maybe
• Driver - No
• Search - Not necessary
• Settings - already done

With slog v1.0.0 and slog-bloat v0.0.0 both activated I got the following on the first display of the Slog admin page.

I believe it was because I hadn't selected a daily trace summary file.

Fatal error: Uncaught Error: Call to a member function fetch_table() on null in /home/customer/www/herbmiller.me/public_html/wp-content/plugins/slog/libs/class-slog-reporter.php:77 Stack trace: #0 /home/customer/www/herbmiller.me/public_html/wp-content/plugins/slog/slog.php(344): Slog_Reporter->fetch_table() #1 /home/customer/www/herbmiller.me/public_html/wp-content/plugins/oik/libs/class-BW-.php(97): slog_admin_table() #2 /home/customer/www/herbmiller.me/public_html/wp-content/plugins/slog/slog.php(107): BW_::oik_box(NULL, 'slog_admin_tabl...', 'CSV table', 'slog_admin_tabl...') #3 /home/customer/www/herbmiller.me/public_html/wp-includes/class-wp-hook.php(287): slog_admin_page('') #4 /home/customer/www/herbmiller.me/public_html/wp-includes/class-wp-hook.php(311): WP_Hook->apply_filters('', Array) #5 /home/customer/www/herbmiller.me/public_html/wp-includes/plugin.php(484): WP_Hook->do_action(Array) #6 /home/customer/www/herbmiller.me/public_html/wp-admin/admin.php(259): do_action('settings_page_s...') #7 /home/customer/ww in /home/customer/www/herbmiller.me/public_html/wp-content/plugins/slog/libs/class-slog-reporter.php on line 77

When attempting to download a file from a remote server, it's possible to get the file name wrong, or find that the file is protected from general access. We need to cater for this.

Warning: file_get_contents(https://cwiccer.com/bwtrace.vt.20210208): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in C:\apache\htdocs\wordpress\wp-content\plugins\slog\libs\class-slog-admin.php on line 385

Back in May 2020 I updated vt.php to calculate very high level averages for the bwtrace.vt.ccyymmdd files for a number of websites, saving the output in a number of .csv files.

What I didn't do was to calculate the average response for each site, nor tabulate it by hosting.

So I'll do that now.

Counting home page requests only.

When running Slog driver to produce comparison files it's currently necessary to

1. Use oik trace options to blank out the trace summary file
2. Get set up prior to running the driver
3. Use oik trace options to set the trace summary file for the test run
4. Run the driver
5. Use oik trace options to blank out the trace summary file
6. Run Slog admin > Compare
7. Repeat for another test.

In a (local) subdirectory install, when producing a report grouped by Stripped request URIs top level it makes more sense to look at the second part of any request that's prefixed by the subdirectory installs folder name.
This should be easy when slog is being run against the local file.

example summary file

/oikcom/?ao_speedup_cachebuster=13906,,6.584218,7.4.0,1293,6355,979,57,1349,74,41,23,168,0.33548879623413,C:/apache/htdocs/oikcom/bwtrace/bwtraces.loh.1,916,0,30985,127.0.0.1,6.547538,2021-01-31T09:04:22+00:00,WordPress/5.6; https://s.b/oikcom,GET,200
/oikcom/wp-admin/admin-ajax.php?action=autoptimize_delete_cache&nonce=baf8fa1366&_=1612083833326,autoptimize_delete_cache,23.861941,7.4.0,1293,6624,962,57,1351,74,41,23,77,,,,,18868,127.0.0.1,23.787242,2021-01-31T09:04:22+00:00,Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML; like Gecko) Chrome/88.0.4324.104 Safari/537.36,GET,200
/oikcom/,,16.533602,7.4.0,1293,6355,975,57,1349,74,41,23,170,0.26832222938538,C:/apache/htdocs/oikcom/bwtrace/bwtraces.loh.9,916,0,30672,127.0.0.1,16.497638,2021-01-31T09:04:26+00:00,Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML; like Gecko) Chrome/88.0.4324.104 Safari/537.36,GET,200
/oikcom/wordpress-plugins-from-oik-plugins/,,3.736265,7.4.0,1293,6167,962,57,1324,74,41,23,169,0.21986675262451,C:/apache/htdocs/oikcom/bwtrace/bwtraces.loh.4,746,1,29184,127.0.0.1,3.705882,2021-01-31T09:04:33+00:00,Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML; like Gecko) Chrome/88.0.4324.104 Safari/537.36,GET,200
/oikcom/wp-cron.php?doing_wp_cron=1612083943.3039140701293945312500,,1.817750,7.4.0,1293,5809,875,57,1191,74,41,23,58,0.073328971862793,C:/apache/htdocs/oikcom/bwtrace/bwtraces.loh.8,129,0,16352,127.0.0.1,1.808337,2021-01-31T09:05:45+00:00,WordPress/5.6; https://s.b/oikcom,POST,200

It's not so easy if the daily trace summary file has been downloaded from another system.

<!-- Cached by WP-Optimize (gzip) - https://getwpo.com - Last modified: Thu, 02 Dec 2021 08:36:42 GMT -->

 
There are some cacheing plugins that deliver cached content to the requester.
In theory, delivering cached content should speed up the website
Slog's Driver tab isn't noticing when the served response has been cached.
There are a couple of problems to be considered.

1. Slog admin > Driver does not appear to get served the cached results
2. If it was server the cache results the Daily Trace Summary would not record the transaction
3. So server performance statisfics would only show the average time of requests actually satisfied
4. It could not be used to summarise requests.
5. It may be necessary to get information from the access log

When changing settings on slog admin > driver I get warning messages from sg-cachepress.
These occur with the latest version of SiteGround Optimizer, the sg-cachepress plugin .

Warning: Use of undefined constant COOKIEHASH - assumed 'COOKIEHASH' (this will throw an Error in a future version of PHP) in /home/customer/www/bobbingwide.com/public_html/wp-content/plugins/sg-cachepress/core/File_Cacher/File_Cacher.php on line 785

Warning: Cannot modify header information - headers already sent by (output started at /home/customer/www/bobbingwide.com/public_html/wp-content/plugins/sg-cachepress/core/File_Cacher/File_Cacher.php:785) in /home/customer/www/bobbingwide.com/public_html/wp-admin/includes/misc.php on line 1416

In addition to displaying a chart showing Count / Total elapsed / Average/ Percentage / etc for the selected groups I'd like to see the calculated data that could have been used to create the chart in the form of a table.
This can be used to decide which is the most useful chart to display the data.

Requirements

• Shows the columns as in the example.
• Column 1 heading to be the report type
• Rows to show the filtered results after applying the Having value ( same as for the chart )

Example

For the Request Types report, with Having set to 2, using /tests/data/request-types-test.vt.20210106

Proposed solution

• Display using table tags
• Partially responsive - with a minimum width defined by WordPress
• No need to round figures
• May need to cater for divide by zero

Potential enhancements

• Ability to choose the column(s) as the data for the chart.
• Ability to compare results with other sets of data.

While analysing the daily trace summary files from oik-plugins.co.uk I noticed a problem with the
Stripped request URIs report. It was producing a load of undefined labels.
Test URL: https://s.b/oikcom/wp-admin/options-general.php?page=slog
Source file: C:/apache/htdocs/wordpress/wp-content/plugins/slog-bloat/working/2021/oikcouk0126.csv

Here the results are displayed using a Horizontal bar chart

Unexpected results

1. All the undefined labels
2. The table of results has too many columns
3. The label is far too long... wen viewed as a Bar chart you may only see one bar.

Expected result

1. Only one label
2. Correct number of columns in the table
3. Label length less than so many characters. Where so many is TBD.

Explanation

The problems are because the URI contains commas. The request being.

"/usr/local/cpanel/3rdparty/wp-toolkit/plib/vendor/wp-cli/wp-cli/php/boot-fs.php --path=/home/oikcouk/public_html --no-color config-settings get --params=DB_CHARSET,DB_NAME,DB_USER,DB_PASSWORD,DB_HOST,DOMAIN_CURRENT_SITE,PATH_CURRENT_SITE,MULTISITE,SUBDOMAIN_INSTALL,WP_AUTO_UPDATE_CORE,WPCACHEHOME,WP_DEBUG,WP_DEBUG_LOG,WP_DEBUG_DISPLAY,SCRIPT_DEBUG,SAVEQUERIES,WP_AUTO_UPDATE_CORE,WP_HOME,WP_SITEURL,CONCATENATE_SCRIPTS,DISALLOW_FILE_EDIT,WP_CACHE_KEY_SALT,DISABLE_WP_CRON --format=json",,0.207512,7.4.14,1749,3182,468,1,441,51,10,5,2,0.011967897415161,,,,5060,127.0.0.1,0.207451,2021-01-29T11:48:25+00:00,,GET

Update code to run without problems in PHP 8.1 and 8.2, with WordPress 6.4.1

See bobbingwide/bobbingwide#103

Warning: Undefined array key "_slog_request_filters" in C:\apache\htdocs\wordpress\wp-content\plugins\slog\libs\class-slog-reports-form.php on line 189

Fatal error: Uncaught TypeError: implode(): Argument #1 ($pieces) must be of type array, string given in C:\apache\htdocs\wordpress\wp-content\plugins\slog\libs\class-slog-reports-form.php:189 Stack trace: #0 C:\apache\htdocs\wordpress\wp-content\plugins\slog\libs\class-slog-reports-form.php(189): implode(',', NULL) #1 C:\apache\htdocs\wordpress\wp-content\plugins\slog\libs\class-slog-reports-form.php(114): Slog_Reports_Form->chart_content() #2 C:\apache\htdocs\wordpress\wp-content\plugins\oik-bwtrace\libs\class-BW-.php(97): Slog_Reports_Form->display_chart() #3 C:\apache\htdocs\wordpress\wp-content\plugins\slog\libs\class-slog-admin.php(174): BW_::oik_box(NULL, 'display_chart', 'Chart', Array) #4 C:\apache\htdocs\wordpress\wp-includes\class-wp-hook.php(324): Slog_Admin->nav_tab_reports('') #5 C:\apache\htdocs\wordpress\wp-includes\class-wp-hook.php(348): WP_Hook->apply_filters('', Array) #6 C:\apache\htdocs\wordpress\wp-includes\plugin.php(517): WP_Hook->do_action(Array) #7 C:\apache\htdocs\wordpress\wp-content\plugins\slog\libs\class-slog-admin.php(133): do_action('slog_nav_tab_re...') #8 C:\apache\htdocs\wordpress\wp-content\plugins\slog\slog.php(168): Slog_Admin->process() #9 C:\apache\htdocs\wordpress\wp-includes\class-wp-hook.php(324): slog_admin_page('') #10 C:\apache\htdocs\wordpress\wp-includes\class-wp-hook.php(348): WP_Hook->apply_filters('', Array) #11 C:\apache\htdocs\wordpress\wp-includes\plugin.php(517): WP_Hook->do_action(Array) #12 C:\apache\htdocs\wordpress\wp-admin\admin.php(259): do_action('settings_page_s...') #13 {main} thrown in C:\apache\htdocs\wordpress\wp-content\plugins\slog\libs\class-slog-reports-form.php on line 189

The Slog admin page has tabs to view reports, compare results etc.
It doesn't yet have the ability to run performance tests.
It would be nice to be able to drive the performance comparison testing directly from the Slog admin page.

Requirements

There would be two methods

1. Run the same query limit times.
2. Run a set of queries from a filtered file, performing up to limit times.

Proposed solution

MVP

• Develop a Driver tab that runs XMLHttpRequests to the server
• Support a loop that will run the request limit times
• Provide visual feedback so show that the loop is working and the results

Optional extras

• Make it easily cancellable
• Prevent the Run button from being pressed multiple times
• Support loading a filtered file to provide the URLs for each request
• Dynamically display the results in a chart
• Allow setting of the daily trace summary file name from the Driver tab

Assumptions, Limitations, Questions

• It'll be easier to write using XMLHttpRequests than fetch().
• I can write it using native JavaScript
• No need to use React / Gutenberg - how possible would that be anyway?
• No need for internationalization.
• Logic only needed for admin.
• It'll be easy to write as plain HTML and JavaScript then convert to WordPress
• Could most of the driver logic be written as ALL JavaScript?
• Is CSS needed to style the output?

Add a new tab that will allow searching of the Daily Trace Summary files for a particular string
be it an IP address, request type or whatever.

Requirements

• Ability to search multiple daily trace summary files
• Display results in the browser
• Allow files to be viewed

Proposed solution

• New Search tab
• Input field for the search string
• Output to include records found, count of occurrences, filenames.

Consider whether or not to search for particular fields OR just do a simple strpos().
Also decide how to choose which files to search.
Multi select list or just the whole shebang.

Requirement

• To be able to display a Horizontal bar chart
• No need to use the chartist.js based pompey_chart logic

Proposed solution

• Add Horizontal bar chart type
• Remove logic to call pompey_chart

In issue #9 the logic was to group the values every 0.05 of a second ( 1/20th ) with a maximum value of 5 seconds.
This was fine for slower servers where most responses ranged between 1.5 and 3 seconds but wasn't any good where the average response was a lot less.
We need to be able to set the interval to every n hundredths of a second and reduce the maximum seconds for the final grouping accordingly.

Note: In the top-12 reports produced in Jan 2016 the displayed selection also had a starting value other than 0.

When visiting the Slog admin page ( s.b/wordpress ) I got two warnings displayed when the fields on the Settings page hadn't been saved.

Expected result

No Warnings

Actual result

Warning: Undefined array key "_slog_downloads_dir" in \libs\class-slog-admin.php on line 48

Warning: Undefined array key "_slog_remote_url" in libs\class-slog-admin.php on line 49

Workaround

Save settings, even if the fields are blank.

The Charts displayed in the Slog admin tabs are produced by chart.js and are interactive.
If you click on the legend then it toggles the display of the line.
This can make it easier to highlight the data you're most interested in looking at.
eg For the top 12 WordPress plugin comparison we might want to focus on the effect of Yoast SEO, WooCommerce and Jetpack.

I want to be able to copy the chart data to the clipboard or a CSV file to be able to display the charts elsewhere
They will have to be sites which are using the sb-chart-block plugin.

Requirement

To be able to download the table as a CSV file.

Proposed solution

• Button to request that the CSV results be copied to the clipboard
• The clipboard can then be pasted to the sb-chart-block.

On a newish site I noticed thousands of requests to wp-login.php from the same IP address. The only thing this site was trying to do was login, using a dictionary lookup for passwords.

Blocking the IP reduced the number of requests to the site to a manageable level.

I realised that it would be easy to detect the IPs that should be blocked simply on the type of requests they perform.

Requirement

1. Routine to analyse the daily summary report to find IPs that should be blocked.
2. Some automated routine to be proactive rather than reactive.

I imagine Jetpack's Protect does something similar. And maybe Wordfence as well.

Is there any point in attempting to write some code that repeats what Lighthouse can do, with regards to measuring your website performance?

How can we measure the "page size" and other values that are calculated by the tools mentioned by Hannah Smith at WP_Hooked tonight?

ecograder.com
ecoping.earth
digitalbeacon.co
websitecarbon.com

What does Query Monitor do?

Earlier today I ran some tests to measure the performance overhead of WooCommerce.

• woocommerce.20211025 are the results for WooCommerce activated
• nowooreally20211025 are the results for oik-unloader deactivating WooCommerce

When compared against results from yesterday noseonotrace.20211024 each invocation took significantly longer.

I realised / guessed this was because I was running my laptop using battery power.
I plugged in and re-ran the tests.

• nowoopower.20211025 are the results when WooCommerce and Yoast SEO are deactivated by oik-unloader
• woopower.20211025 shows the results when oik-unloader is not active; both WooCommerce and Yoast SEO loaded

While filtering daily trace summary files to create files to drive the performance analysis routines on a remote server
I find myself toggling between slog and slog-bloat. Sometimes I want to see the request types in the ( downloaded ) daily trace summary file.
It would be nice if the charts/reports in slog were easily accessible from slog-bloat.

Requirement

• Ability to run slog reports against filtered / downloaded files
• Make slog and slog bloat easier to use
• Reduce complexity

Current position

• Both slog and slog bloat are dependent upon oik-bwtrace and sb-chart-block.
• Both use shared library functions
• slog's admin was written as procedural code
• slog-bloat's admin is better structured
• slog-bloat is dependent upon slog
• slog-bloat relies on slog's oik_autocode logic - is this true?

Proposed solution

• Extend slog's file list to include files in slog-bloat's downloaded directory ( slog_bloat_options[_slog_downloads_dir] )
• Cater for the Download directory not being specified, or blank.

I've decided this is a sensible idea.

The final outcome being that:

• slog will do the charting stuff
• either slog-bloat will contain the oik-batch vt-driver.php logic or it's merged into slog.
• slog-bloat will retain the historical results.

Steps to achieving this:

• Refactor the slog report logic to be object oriented as a Reports tab under slog-bloat.
• Rework the settings logic so that the options are only updated on the Settings tab.
• Remove slog-bloat's dependency on slog.
• When the refactoring is complete copy slog-bloat logic to slog.
• Eliminate admin logic from slog-bloat

The routines developed to determine the performance impact of the Top 12 WordPress plugins should be published as samples then modified over time to create the general purpose routines that others may use.

Q. Can this be achieved using branches alone or should it be done with a different repository?
A. We'll use http://GitHub.com/bobbingwide/wp-top12 for the code built to analyse the top 12 WordPress plugins

For slog-bloat I need the figures for each file being compared and need to know how many requests were dealt with in each interval of 1/20th of a second ( 0.05 secs ).
Otherwise the chart goes well wrong.
It's quite pretty... but useless.

1. Sort sequence on the x-axis is wrong
2. Labels are wrong
3. Values shouldn't drop to 0 for accumulated percentages!

These changes also apply to the Elapsed chart displayed in slog.

It should be possible to run slog and slog-bloat without wp-top12.

Requirement

slog and slog-bloat to be mostly standalone

Proposed solution

• Implement oik_autoload
• Remove calls to oik_require for files from wp-top12
• Convert slog_admin_* functions to class Slog_Admin

After a while there is a build up of files named bwtrace.vt.yymmdd and, for multisite additional files bwtrace.vy.yymmdd.id. It would be nice to have some backend logic to help manage them.

Once the files have been safely extracted from the server they are no longer needed, so a delete facility would be nice. It would require an admin interface and/or an AJAX/REST one as we need to restrict processing to authorised users.

A while ago, the logic written in vt.php was copied and cobbled into wp-top12.
It was extended as vt-stats.php and vt-top12.php.
Both routines produce CSV files summarising daily trace output in a variety of ways.
See the readme file for https://github.com/bobbingwide/wp-top12

I now want to convert the batch routines into Admin pages that extend oik-bwtrace.
and provide a variety of charts to be displayed in the Dashboard.
The admin pages will allow the user to select the required analysis.

Requirement

• Ability to analyse the Daily Trace Summary reports in the WordPress dashboard
• Choose the file or files to be analysed.
• Choose from a range of reports

Proposed solution

• Add an admin page
• Support WordPress Multi Site summary trace files
• Use/extend code from files in wp-top12 and pompey-chart
• Eventually the code will have to be converted to shared libraries.

The admin page will display:

• A form for input file and chart selection.
• Selected chart
• The chart's raw data in tabular form.

Form fields

Reports