Code Monkey home page Code Monkey logo

database's Introduction

The Database Package Build Status Build status

Latest Stable Version Total Downloads Latest Unstable Version License

Introduction

The Database package is designed to manage the operations of data management through the use of a generic database engine.

// Example for initialising a database driver in a custom application class.

use Joomla\Application\AbstractApplication;
use Joomla\Database;

class MyApplication extends AbstractApplication
{
	/**
	 * Database driver.
	 *
	 * @var    Database\DatabaseDriver
	 * @since  1.0
	 */
	protected $db;

	protected function doExecute()
	{
		// Do stuff
	}

	protected function initialise()
	{
		// Make the database driver.
		$dbFactory = new Database\DatabaseFactory;

		$this->db = $dbFactory->getDriver(
			$this->get('database.driver'),
			array(
				'host' => $this->get('database.host'),
				'user' => $this->get('database.user'),
				'password' => $this->get('database.password'),
				'port' => $this->get('database.port'),
				'socket' => $this->get('database.socket'),
				'database' => $this->get('database.name'),
			)
		);
	}
}

Escaping Strings and Input

Strings must be escaped before using them in queries (never trust any variable input, even if it comes from a previous database query from your own data source). This can be done using the escape and the quote method.

The escape method will generally backslash unsafe characters (unually quote characters but it depends on the database engine). It also allows for optional escaping of additional characters (such as the underscore or percent when used in conjunction with a LIKE clause).

The quote method will escape a string and wrap it in quotes, however, the escaping can be turned off which is desirable in some situations. The quote method will also accept an array of strings and return an array quoted and escaped (unless turned off) string.

function search($title)
{
	// Get the database driver from the factory, or by some other suitable means.
	$db = DatabaseDriver::getInstance($options);

	// Search for an exact match of the title, correctly sanitising the untrusted input.
	$sql1 = 'SELECT * FROM #__content WHERE title = ' . $db->quote($title);

	// Special treatment for a LIKE clause.
	$search = $db->quote($db->escape($title, true) . '%', false);
	$sql2 = 'SELECT * FROM #__content WHERE title LIKE ' . $search;

	if (is_array($title))
	{
		$sql3 = 'SELECT * FROM #__content WHERE title IN ('
			. implode(',', $db->quote($title)) . ')';
	}

	// Do the database calls.
}

In the first case, the title variable is simply escaped and quoted. Any quote characters in the title string will be prepended with a backslash and the whole string will be wrapped in quotes.

In the second case, the example shows how to treat a search string that will be used in a LIKE clause. In this case, the title variable is manually escaped using escape with a second argument of true. This will force other special characters to be escaped (otherwise you could set youself up for serious performance problems if the user includes too many wildcards). Then, the result is passed to the quote method but escaping is turned off (because it has already been done manually).

In the third case, the title variable is an array so the whole array can be passed to the quote method (this saves using a closure and a )

Shorthand versions are available the these methods:

  • q can be used instead of quote
  • qn can be used instead of quoteName
  • e can be used instead of escape

These shorthand versions are also available when using the Database\DatabaseQuery class.

Iterating Over Results

The Database\DatabaseIterator class allows iteration over database results

$db = DatabaseDriver::getInstance($options);
$iterator = $db->setQuery(
	$db->getQuery(true)->select('*')->from('#__content')
)->getIterator();

foreach ($iterator as $row)
{
    // Deal with $row
}

It allows also to count the results.

$count = count($iterator);

Logging

Database\DatabaseDriver implements the Psr\Log\LoggerAwareInterface so is ready for intergrating with a logging package that supports that standard.

Drivers log all errors with a log level of LogLevel::ERROR.

If debugging is enabled (using setDebug(true)), all queries are logged with a log level of LogLevel::DEBUG. The context of the log include:

  • sql : The query that was executed.
  • category : A value of "databasequery" is used.

An example to log error by Monolog

Add this to composer.json

{
	"require" : {
		"monolog/monolog" : "1.*"
	}
}

Then we push Monolog into Database instance.

use Monolog\Logger;
use Monolog\Handler\StreamHandler;
use Monolog\Processor\PsrLogMessageProcessor;

// Create logger object
$logger = new Logger('sql');

// Push logger handler, use DEBUG level that we can log all information
$logger->pushHandler(new StreamHandler('path/to/log/sql.log', Logger::DEBUG));

// Use PSR-3 logger processor that we can replace {sql} with context like array('sql' => 'XXX')
$logger->pushProcessor(new PsrLogMessageProcessor);

// Push into DB
$db->setLogger($logger);
$db->setDebug(true);

// Do something
$db->setQuery('A WRONG QUERY')->execute();

This is the log file:

[2014-07-29 07:25:22] sql.DEBUG: A WRONG QUERY {"sql":"A WRONG QUERY","category":"databasequery","trace":[...]} []
[2014-07-29 07:36:01] sql.ERROR: Database query failed (error #42000): SQL: 42000, 1064, You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'A WRONG QUERY' at line 1 {"code":42000,"message":"SQL: 42000, 1064, You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'A WRONG QUERY' at line 1"} []

Installation via Composer

Add "joomla/database": "~2.0" to the require block in your composer.json and then run composer install.

{
	"require": {
		"joomla/database": "~2.0"
	}
}

Alternatively, you can simply run the following from the command line:

composer require joomla/database "~2.0"

If you want to include the test sources, use

composer require --prefer-source joomla/database "~2.0"

database's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

database's Issues

Postgresql Driver Execution errors on insert

The Postgresql Driver has Execution errors on insert with the following the unit tests on travis

1) Joomla\Database\Tests\DriverPostgresqlTest::testExecute
Joomla\Database\Exception\ExecutionFailureException: 
SQL=INSERT INTO jos_dbtest
(title,start_date,description) VALUES 
('testTitle','1970-01-01','testDescription')
/database/src/Postgresql/PostgresqlDriver.php:819
/database/Tests/DriverPostgresqlTest.php:1131

2) Joomla\Database\Tests\DriverPostgresqlTest::testExecutePreparedStatement
Joomla\Database\Exception\ExecutionFailureException: 
SQL=INSERT INTO jos_dbtest
(title,start_date,description) VALUES 
($1, $2, $3)
/database/src/Postgresql/PostgresqlDriver.php:819
/database/Tests/DriverPostgresqlTest.php:1160

database model

As a database specialist I sometimes manipulate data directly in the joomla database. Every time I do so, I'm frustrated by the joomla data model. It is complex, redundant, allows invalid data. It looks like a datamodel which hasn't been built by concept, but grown wildly.
I'm sure programming (and data management) would be much easier and less buggy with a proper data model.
I suggest to bring several database specialists together in order to simplyfy the joomla data model for the next major release. I'm ready to work in such a group if I'm needed.

Mysqli Iterator fails unit tests

The Mysqli Iterator fails the unit tests. Off hand from the failures, it seems there may be an indexing issue adding an additional row to the arrays.

There were 4 failures:

1) Joomla\Database\Tests\IteratorMysqliTest::testForEach with data set #0 ('title', '#__dbtest', null, 'stdClass', 0, 0, array(stdClass Object (...), stdClass Object (...), stdClass Object (...), stdClass Object (...)), null)
136
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => stdClass Object (...)
     1 => stdClass Object (...)
     2 => stdClass Object (...)
     3 => stdClass Object (...)
+    4 => stdClass Object (...)
 )
/database/Tests/IteratorMysqliTest.php:137

2) Joomla\Database\Tests\IteratorMysqliTest::testForEach with data set #2 ('title', '#__dbtest', null, 'stdClass', 20, 2, array(stdClass Object (...), stdClass Object (...)), null)
136
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => stdClass Object (...)
     1 => stdClass Object (...)
+    2 => stdClass Object (...)
 )
/database/Tests/IteratorMysqliTest.php:137

3) Joomla\Database\Tests\IteratorMysqliTest::testForEach with data set #3 ('title, id', '#__dbtest', 'title', 'stdClass', 0, 0, array(stdClass Object (...), stdClass Object (...), stdClass Object (...), stdClass Object (...)), null)
136
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     'Testing' => stdClass Object (...)
     'Testing2' => stdClass Object (...)
     'Testing3' => stdClass Object (...)
     'Testing4' => stdClass Object (...)
+    'testRollback' => stdClass Object (...)
 )
/database/Tests/IteratorMysqliTest.php:137

4) Joomla\Database\Tests\IteratorMysqliTest::testCount
153
Failed asserting that 5 matches expected 4.
/database/Tests/IteratorMysqliTest.php:154

[RFC] Reduce the number of AppVeyor jobs

AppVeyor's free option only allows for 1 concurrent job, this can create long wait times for builds as they are stuck in the Queue.

I suggest that we choose to only test php 5.6 or php 7.1 on AppVeyor to reduce the overall wait on builds (I would preference php 7.1)

Another option is to convince the PLT to pay for AppVeyor's service to get faster builds with more concurrent jobs.

The last option is to just accept the slow build window and the long Queue waits.

add MSSQL testing via Travis CI and docker images

Steps to reproduce the issue

no continuous integration testing is currently performed on some database platforms like MSSQL, etc

Expected result

Test against all our database platforms like MSSQL, etc

Actual result

no continuous integration testing is currently performed

Additional comments

I recently read the following

The SQL Server for Linux preview is available as a docker image. Assuming we can setup Travis to protect our access key to the private preview, we could use docker on Travis CI builds to execute against tests SQL Server.

https://www.microsoft.com/en-us/server-cloud/sql-server-on-linux.aspx

https://docs.travis-ci.com/user/docker/

(I am aware that the CMS 4.x is dropping MSSQL and that the Framework may follow at some point due to low usage and lack of CI testing)

union query failed to implement

Steps to reproduce the issue

Union queries do not achieve the desired effect

Expected result

SELECT c.dataindex FROM `simpledatacenter` AS `c` WHERE c.datatype = 'downlink' AND c.dataindex !='' AND c.update_time >'2017-12-13 03:16:29'
UNION
SELECT b.sign FROM `simpledatacenter` AS `b` WHERE b.datatype = 'official' AND b.update_time >'2017-12-13 03:16:06' AND ( SELECT count(*) FROM `simpledatacenter` AS `a` WHERE a.datatype = 'downlink' AND a.dataindex = b.sign) > 0

Actual result

SELECT c.dataindex FROM `simpledatacenter` AS `c` WHERE c.datatype = 'downlink' AND c.dataindex !='' AND c.update_time >'2017-12-13 03:42:39'

System information (as much as possible)

joomla-framework/database Newest

Additional comments

I'm using a database connected by mysqli; It is hoped that the document can be supplemented.
The above content comes from software translation.

        $last_update_time = date('Y-m-d H:i:s', time() - 60);
        $client = $this->StorageInstance->getClient();

        $subQuery = $client->getQuery(true);
        $subQuery->from($client->quoteName($this->StorageInstance->getTable(), 'a'));
        $subQuery->select('count(*)');
        $subQuery->where('a.datatype = ' . $client->quote('downlink'));
        $subQuery->where('a.dataindex = b.sign');

        $unionQuery = $client->getQuery(true);
        $unionQuery->from($client->quoteName($this->StorageInstance->getTable(), 'b'));
        $unionQuery->select('b.sign');
        $unionQuery->where('b.datatype = ' . $client->quote('official'));
        $unionQuery->where('b.update_time >' . $client->quote($last_update_time));
        $unionQuery->where('(' . $subQuery . ') > 0');

        $mainQuery = $client->getQuery(true);
        $mainQuery->from($client->quoteName($this->StorageInstance->getTable(), 'c'));
        $mainQuery->select('c.dataindex');
        $mainQuery->where('c.datatype = ' . $client->quote('downlink'));
        $mainQuery->where('c.dataindex !=' . $client->quote(''));
        $mainQuery->where('c.update_time >' . $client->quote($last_update_time));
        $mainQuery->union($unionQuery);

        var_dump($mainQuery->__toString());
        exit();

[2.0] Unsupported ROW_NUMBER() function

public function selectRowNumber($orderBy, $orderColumnAlias)

The function ROW_NUMBER() seems to be introduced in MySQL 8.
https://www.mysqltutorial.org/mysql-window-functions/mysql-row_number-function/

Though Joomla 4 seems to require MySQL 5.6 as a minimum.
https://developer.joomla.org/news/788-joomla-4-on-the-move.html

Is that feature checked against MySQL version lower to 8 ?

This maybe related with issues regarding the records ordering in the database tables.
e.g. joomla/joomla-cms#23905

For versions lower to 8 it suggests ways to emulate it.
https://www.mysqltutorial.org/mysql-row_number/

[2.0] Rewrite Test Suite for better functional test scenarios

Our test suite tries to unit test a lot of the basic things just to prove functionality but misses proper functional test coverage of much of the API. Also, the current suite structure requires duplicating a lot of the test cases. Look for a better way to structure the test suite and rewrite to better test the package in general.

Offset and limit set two times

Is there a chance to have limit set two times for a query ?
Example :

$db    = $this->getDb();
$query = $db->getQuery(true);

$query->select('a.id')
            ->from('#__tests AS a')
            ->where('a.id > 10');

$db->setQuery($query, 0, 1);
$result = $db->loadResult();

In MysqliDriver::setQuery() we have this :

if ($query instanceof LimitableInterface && !is_null($offset) && !is_null($limit))
{
    $query->setLimit($limit, $offset);
}

and then, in MysqliDriver::execute() we have :

if ($this->limit > 0 || $this->offset > 0)
{
    $sql .= ' LIMIT ' . $this->offset . ', ' . $this->limit;
}

So, my query looks like that :

SELECT a.id FROM abc_tests WHERE a.id > 10 LIMIT 1 LIMIT 0, 1

Postgresql / PDO Postgresql Driver issues on Windows

Steps to reproduce the issue

Review AppVeyor tests in PR #72

Expected result

no Postgresql / PDO Postgresql driver issues on windows

Actual result

the following failures occurred with GetCollation() and GetVersion()

There were 4 failures:
1) Joomla\Database\Tests\DriverPgsqlTest::testGetCollation
224
Failed asserting that 'English_United States.1252' contains "UTF-8".
C:\projects\database\Tests\DriverPgsqlTest.php:224
2) Joomla\Database\Tests\DriverPgsqlTest::testGetVersion
465
Failed asserting that '9.4.7' is equal to <string:9.4.7,> or is greater than '9.4.7,'.
C:\projects\database\Tests\DriverPgsqlTest.php:465
3) Joomla\Database\Tests\DriverPostgresqlTest::testGetCollation
224
Failed asserting that 'English_United States.1252' contains "UTF-8".
C:\projects\database\Tests\DriverPostgresqlTest.php:224
4) Joomla\Database\Tests\DriverPostgresqlTest::testGetVersion
465
Failed asserting that '9.4.7' is equal to <string:9.4.7,> or is greater than '9.4.7,'.
C:\projects\database\Tests\DriverPostgresqlTest.php:465

Additional comments

n/a

[2.0-dev] Uncaught Joomla\Database\Exception\PrepareStatementFailureException: Unknown column

Steps to reproduce the issue

Create a query with column value like SELECT 'hello';

print_r($db->setQuery($db->getQuery(true)->select('ala'))->loadColumn());
PHP Warning:  Uncaught Joomla\Database\Exception\PrepareStatementFailureException: Unknown column 'ala' in 'field list' in /home/tomash/public_html/database/src/Mysqli/MysqliStatement.php:124
Stack trace:
#0 /home/tomash/public_html/database/src/Mysqli/MysqliDriver.php(860): Joomla\Database\Mysqli\MysqliStatement->__construct(Object(mysqli), 'SELECT ala')
#1 /home/tomash/public_html/database/src/DatabaseDriver.php(1755): Joomla\Database\Mysqli\MysqliDriver->prepareStatement('SELECT ala')
#2 php shell code(1): Joomla\Database\DatabaseDriver->setQuery(Object(Joomla\Database\Mysqli\MysqliQuery))
#3 {main}
  thrown in /home/tomash/public_html/database/src/Mysqli/MysqliStatement.php on line 124

Expected result

Should work.

System information (as much as possible)

2.0-dev

Additional comments

I tested it by php -a and use require '../vendor/autoload.php';

CI configuration unexpectedly leaks into testing environment

Commits such as 073aff4 have leaked environment variables used to configure the various CI platforms into the test suite. This essentially changes the testing workflow so that it will only run correctly on a properly configured CI platform, different from the user expectation that the tests should be executable in any environment with the correct settings.

If it is intended to require this type of configuration when executing the tests, these requirements should follow existing conventions for defining constants or env vars and clearly documented for all contributors to use. Otherwise, not documenting this mandatory change will result in unexpected behaviors if the tests are run outside of the configured CI environment.

Is it possible to bind null values?

Tried to bind null values in UPDATE queries with little luck.

Using PHP's null:

$value = null;
$query->bind(':key', $value, ParameterType::NULL);

MySQL: No data supplied for parameters in prepared statement

PDO: HY093, ,

PostgreSQL: ERROR: bind message supplies 0 parameters, but prepared statement "pdo_stmt_00000034" requires 1

Using NULL string:

$value = 'NULL';
$query->bind(':key', $value, ParameterType::NULL);

MySQLi: Incorrect datetime value: 'NULL'. Using ParameterType::INTEGER doesn't error but then getting the same result as PDO.

PDO: No errors but value set is incorrect, e.g. on datetime column with NULL as default it sets 0000-00-00 00:00:00 value.

PostgreSQL: works OK, I think.

Need consistent syntax for parameterized query support

Starting with the 1.5.0 tag, all drivers have support for prepared statements and parameterized queries, and in 2.0 this will be a hard requirement because of the new QueryInterface dictating it. However, the current implementation is dependent on the syntax(es) supported by the underlying driver, meaning at the moment it isn't possible to actually write "properly" abstract queries with parameterized query support (unless an application is only supporting PDO + MySQLi)

  • Question mark syntax (INSERT INTO foo (col1, col2) VALUES (?, ?)) is supported by PDO, MySQLi, and SQL Server
  • Named parameter syntax (INSERT INTO foo (col1, col2) VALUES (:col1, :col2)) is supported by PDO
  • "PHP variable" syntax (INSERT INTO foo (col1, col2) VALUES ($1, $2)) is supported by PostgreSQL (ext/pgsql, separate from the PDO driver)

So in essence we need some kind of SQL parsing handler in the DatabaseQuery class. My thought was to have it support the question mark syntax as that one is already supported by most drivers and we only need a PostgreSQL implementation. While discussing in chat @nibra suggested we standardize around the named parameter syntax since it creates an easier to read and parse string.

Whatever the end result, we do need a solution that allows users of our API to not be concerned with these driver specific details (it's OK for now if you're writing single database applications, but because the CMS is using this package in 4.0, it becomes a problem).

DatabaseQuery::union BUG?

I see the to string method looking to see if DatabaseQuery::$type is equal to union but it doesn't get set anywhere in the class is this a bug or am I trying to do it wrong?

[2.0] Deprecate LimitableInterface

The LimitableInterface should be deprecated and its logic moved to the QueryInterface. All of our query objects (finally) support setting query limits on the query object so this should be the preferred API for setting limits versus setting them via the database driver and setQuery().

[RFC] Adopt stricter MySQL sql_mode?

in Joomla 4.0 joomla used a strict sql mode
https://github.com/joomla/joomla-cms/pull/12494/files

This was reverted when joomla 4.0 was updated to use the framework database package.

As we know MySql is becoming stricter as the other database systems (postgresql, Mssql) already are.

Example:

In MySQL 5.7.4, the ERROR_FOR_DIVISION_BY_ZERO, NO_ZERO_DATE, and NO_ZERO_IN_DATE SQL modes were changed so that they did nothing when named explicitly. Instead, their effects were included in the effects of strict SQL mode (STRICT_ALL_TABLES or STRICT_TRANS_TABLES). The intent was to reduce the number of SQL modes with an effect dependent on strict mode and make them part of strict mode itself.

However, the change to make strict mode more strict by including ERROR_FOR_DIVISION_BY_ZERO, NO_ZERO_DATE, and NO_ZERO_IN_DATE caused some problems. For example, in MySQL 5.6 with strict mode but not NO_ZERO_DATE enabled, TIMESTAMP columns can be defined with DEFAULT '0000-00-00 00:00:00'. In MySQL 5.7.4 with the same mode settings, strict mode includes the effect of NO_ZERO_DATE and TIMESTAMP columns cannot be defined with DEFAULT '0000-00-00 00:00:00'. This causes replication of CREATE TABLE statements from 5.6 to 5.7.4 to fail if they contain such TIMESTAMP columns.

The long term plan is still to have the three affected modes be included in strict SQL mode and to remove them as explicit modes in a future MySQL release. But to restore compatibility in MySQL 5.7 with MySQL 5.6 strict mode and to provide additional time for affected applications to be modified, the following changes have been made:
[...]

See https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-8.html#mysqld-5-7-8-sql-mode

Also see discussion in joomla/joomla-cms#16788

[2.0] PdoDriver with junk in code

Steps to reproduce the issue

Please remove junk from the merge conflict.

<<<<<<< HEAD
=======
* Sets the SQL statement string for later execution.
*
* @param mixed $query The SQL statement to set either as a JDatabaseQuery object or a string.
* @param integer $offset The affected row offset to set.
* @param integer $limit The maximum affected rows to set.
* @param array $driverOptions The optional PDO driver options
*
* @return PdoDriver This object to support method chaining.
*
* @since 1.0
*/
public function setQuery($query, $offset = null, $limit = null, $driverOptions = array())
{
$this->connect();
$this->freeResult();
if (\is_string($query))
{
// Allows taking advantage of bound variables in a direct query:
$query = $this->getQuery(true)->setQuery($query);
}
if ($query instanceof LimitableInterface && !\is_null($offset) && !\is_null($limit))
{
$query->setLimit($limit, $offset);
}
$sql = $this->replacePrefix((string) $query);
$this->prepared = $this->connection->prepare($sql, $driverOptions);
// Store reference to the DatabaseQuery instance:
parent::setQuery($query, $offset, $limit);
return $this;
}
/**
>>>>>>> 485053508fc83aa8c9deaefb7e27efecf42a85e8

The MySQL version check for integer display widths modifier support is not accurate enough

Steps to reproduce the issue

The following and possibly more changes might not be sufficient regarding the version check:

073aff4#diff-3e927448d9eaafa620c1a995e71ad8d5ab1cae1e56ef4b6a2bd78d68a430ae56R306

ab96e2a#diff-45e47bfc8339fb2db89de46a41ff0b70379e2b570c44ca9685899d898deb099cR317

MySQL who obviously don't use semver (at least not for 8.0.x) have deprecated the display width modifier beginning with version 8.0.17, see https://dev.mysql.com/doc/refman/8.0/en/numeric-type-attributes.html .

I remember from past when I had discovered that after my MSQL had been updated so I had to make PR joomla/joomla-cms#28501 , the display width modifier was part of the type on the 8.0.x version I had before that update.

That means the version check here should be done against version 8.0.17, and that can't be done with a floating point type cast and comparison, it needs to use something like VERSION_COMPARE for that.

Furthermore, MariaDB still support the display width modifier and so behaves like a MySQL < 8.0.17 regarding this. The version check should check this, too.

=> Ping @nibra .

Can't bind limit parameters using MySQLi driver

Steps to reproduce the issue

Sample code in CMS:

use Joomla\CMS\Factory;
use Joomla\Database\ParameterType;

$limit = 5;
$db = Factory::getDbo();
$query = $db->getQuery(true)
	->select('*')
	->from($db->quoteName('#__content'))
	->setLimit(':limit')
	->bind(':limit', $limit, ParameterType::INTEGER);
$results = $db->setQuery($query)->loadObjectList();

Expected result

5 items returned

Actual result

Error without message. Stack trace:

Call stack
--
# | Function | Location
1 | () | JROOT\libraries\vendor\joomla\database\src\Mysqli\MysqliStatement.php:432
2 | Joomla\Database\Mysqli\MysqliStatement->execute() | JROOT\libraries\vendor\joomla\database\src\DatabaseDriver.php:673
3 | Joomla\Database\DatabaseDriver->execute() | JROOT\libraries\vendor\joomla\database\src\DatabaseDriver.php:1336
4 | Joomla\Database\DatabaseDriver->loadObjectList() | JROOT\templates\cassiopeia\index.php:26
5 | require() | JROOT\libraries\src\Document\HtmlDocument.php:671
6 | Joomla\CMS\Document\HtmlDocument->_loadTemplate() | JROOT\libraries\src\Document\HtmlDocument.php:733
7 | Joomla\CMS\Document\HtmlDocument->_fetchTemplate() | JROOT\libraries\src\Document\HtmlDocument.php:543
8 | Joomla\CMS\Document\HtmlDocument->parse() | JROOT\libraries\src\Application\CMSApplication.php:956
9 | Joomla\CMS\Application\CMSApplication->render() | JROOT\libraries\src\Application\SiteApplication.php:754
10 | Joomla\CMS\Application\SiteApplication->render() | JROOT\libraries\src\Application\CMSApplication.php:247
11 | Joomla\CMS\Application\CMSApplication->execute() | JROOT\includes\app.php:63
12 | require_once() | JROOT\index.php:36

System information (as much as possible)

Joomla! CMS 4.0-dev nightly, MariaDB 10.3

Additional comments

This works fine using PDO driver.

MySQLi can't load custom class objects if they contain protected properties

Steps to reproduce the issue

Do $db->loadObjectList('key', SomeClass::class) when SomeClass contains protected properties.

Expected result

Works.

Actual result

Trying to access protected property error.

Additional comments

I'm not sure whether this is expected or not because this works fine on PDO drivers.

[RFC] Deprecate some alias functions in DatabaseQuery

In the CMS we moved our method calls from join('left') to leftJoin() I'm not sure if this really makes sense because it's only a alias function to join() so we have small overhead.

Also it makes the API more complex and less maintainable.
Most of the functions are not in the interface which seams to make a problem in IntelliJ IDE so if
we don't remove them they should be in the interface.

Functions for discussion:
[]Join functions
[
]where functions

Functions missing in the interface
dateAdd
dateFormat
dump
escape (and alias)
quote (and alias)
quoteName (and alias)
format

Is there a reason why these functions are not in the interface?

DatabaseDriver function `__call()` docblock has wrong return type

Steps to reproduce the issue

the DatabaseDriver function __call() docblock return type says

// @return  string  The aliased method's return value or null.

The type indicated here is string

but the comment indicates the type could be null

Additionally, if there are no arguments passed we have a condition returning void
see lines 452-455

Expected result

return type should not error on Travis

Actual result

Function return type is not void, but the function is returning void on line 454

Additional comments

The return type should be adjusted to reflect the correct return type.

Offhand I'm not sure if the docblock comment should be changed to

// @return  string|null|void  The aliased method's return value or null.

or if the return on line 454 should be specified null with the docblock comment changed to

// @return  string|null  The aliased method's return value or null.

Please advise for PR to correct this issue

change the function that calculate the number of items in pagination system

When i have many records the Model system call in libraries/src/MVC/Model/ListModel.php the function getTotal() and this call _getListQuery(), this use the getListQuery() in my component model, remove all limit and make a count(*) of all possible records, so the library can calculate the number of pages.
But if in my model in getListQuery() it's a query with many join, the query that keep the first 20 items (for example) is not a problem, the problem is the count(*) for all records for the pagination.
So the performances are terrible if the items are so many and even the joins.

My solution is add a function in libraries/src/MVC/Model/ListModel.php:

protected function _getPaginationListQuery() {
static $lastStoreId;
$currentStoreId = $this->getStoreId();

    if ($lastStoreId != $currentStoreId || empty($this->query)) {
        $lastStoreId = $currentStoreId;
        if (method_exists($this, "getListPaginationQuery")) {
            $this->query = $this->getListPaginationQuery();
        } else {
            $this->query = $this->getListQuery();
        }
    }
    return $this->query;
}

and change the getTotal in:
public function getTotal() {
// Get a storage key.
$store = $this->getStoreId('getTotal');

    // Try to load the data from internal storage.
    if (isset($this->cache[$store])) {
        return $this->cache[$store];
    }

    try {
        // Load the total and add the total to the internal cache.
        $this->cache[$store] = (int) $this->_getListCount($this->**_getPaginationListQuery()**);
    } catch (\RuntimeException $e) {
        $this->setError($e->getMessage());

        return false;
    }

    return $this->cache[$store];
}

So i can add in my model the function getListPaginationQuery() and insert another query without join or other options that increase the complexity of the same query.
The query in getListPaginationQuery() give the same count of items, but it's best for performance.
(I remove all the Left Join for example or some columns).
If not exist getListPaginationQuery() the library use the standard getListQuery() in component model.

Can this be insert in the next release? thanks.

Database::getTableCreate with db prefixes

Database::getTableCreate returns statement with prefixes instead of placeholder #__:

CREATE TABLE `foo_profile` (

Tested on MysqliDriver, but likely same results for other drivers where the method is present: SqliteDriver, OracleDriver, MysqlDriver

Upmerge of recently merged PRs from master to 2.0-dev

Steps to reproduce the issue

Check if the 2.0-dev branch contains the changes from following PRs: #217 , #219 , #220 .

Expected result

It does.

Actual result

It doesn't.

System information (as much as possible)

Not relevant here.

Additional comments

It seems to be a while ago when the 2.0-dev branch was updated with changes from the master branch.

MSSQL DELETE_ALL tests fail on appveyor see PR #72

Steps to reproduce the issue

MSSQL DELETE_ALL tests fail on appveyor see PR #72

Expected result

tests pass

Actual result

tests fail

There were 36 errors:
1) Joomla\Database\Tests\DriverSqlsrvTest::test__destruct
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
2) Joomla\Database\Tests\DriverSqlsrvTest::testConnected
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
3) Joomla\Database\Tests\DriverSqlsrvTest::testDropTable
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
4) Joomla\Database\Tests\DriverSqlsrvTest::testEscape with data set #0 (''%_abc123', false, '''%_abc123')
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
5) Joomla\Database\Tests\DriverSqlsrvTest::testEscape with data set #1 (''%_abc123', true, '''%[_]abc123')
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
6) Joomla\Database\Tests\DriverSqlsrvTest::testGetAffectedRows
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
7) Joomla\Database\Tests\DriverSqlsrvTest::testGetCollation
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
8) Joomla\Database\Tests\DriverSqlsrvTest::testGetExporter
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
9) Joomla\Database\Tests\DriverSqlsrvTest::testGetImporter
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
10) Joomla\Database\Tests\DriverSqlsrvTest::testGetNumRows
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
11) Joomla\Database\Tests\DriverSqlsrvTest::testGetTableCreate
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
12) Joomla\Database\Tests\DriverSqlsrvTest::testGetTableColumns
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
13) Joomla\Database\Tests\DriverSqlsrvTest::testGetTableKeys
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
14) Joomla\Database\Tests\DriverSqlsrvTest::testGetTableList
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
15) Joomla\Database\Tests\DriverSqlsrvTest::testGetVersion
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
16) Joomla\Database\Tests\DriverSqlsrvTest::testInsertid
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
17) Joomla\Database\Tests\DriverSqlsrvTest::testLoadAssoc
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
18) Joomla\Database\Tests\DriverSqlsrvTest::testLoadAssocList
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
19) Joomla\Database\Tests\DriverSqlsrvTest::testLoadColumn
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
20) Joomla\Database\Tests\DriverSqlsrvTest::testLoadObject
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
21) Joomla\Database\Tests\DriverSqlsrvTest::testLoadObjectList
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
22) Joomla\Database\Tests\DriverSqlsrvTest::testLoadResult
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
23) Joomla\Database\Tests\DriverSqlsrvTest::testLoadRow
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
24) Joomla\Database\Tests\DriverSqlsrvTest::testLoadRowList
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
25) Joomla\Database\Tests\DriverSqlsrvTest::testExecute
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
26) Joomla\Database\Tests\DriverSqlsrvTest::testRenameTable
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
27) Joomla\Database\Tests\DriverSqlsrvTest::testSelect
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
28) Joomla\Database\Tests\DriverSqlsrvTest::testSetUtf
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
29) Joomla\Database\Tests\DriverSqlsrvTest::testIsSupported
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
30) Joomla\Database\Tests\DriverSqlsrvTest::testUpdateObject
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
31) Joomla\Database\Tests\IteratorSqlsrvTest::testForEach with data set #0 ('title', '#__dbtest', null, 'stdClass', 0, 0, array(stdClass Object (...), stdClass Object (...), stdClass Object (...), stdClass Object (...)), null)
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
32) Joomla\Database\Tests\IteratorSqlsrvTest::testForEach with data set #1 ('title', '#__dbtest', null, 'stdClass', 2, 0, array(stdClass Object (...), stdClass Object (...)), null)
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
33) Joomla\Database\Tests\IteratorSqlsrvTest::testForEach with data set #2 ('title', '#__dbtest', null, 'stdClass', 20, 2, array(stdClass Object (...), stdClass Object (...)), null)
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
34) Joomla\Database\Tests\IteratorSqlsrvTest::testForEach with data set #3 ('title, id', '#__dbtest', 'title', 'stdClass', 0, 0, array(stdClass Object (...), stdClass Object (...), stdClass Object (...), stdClass Object (...)), null)
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
35) Joomla\Database\Tests\IteratorSqlsrvTest::testForEach with data set #4 ('title', '#__dbtest', 'title', 'UnexistingClass', 0, 0, array(), 'InvalidArgumentException')
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218
36) Joomla\Database\Tests\IteratorSqlsrvTest::testCount
PHPUnit_Extensions_Database_Operation_Exception: COMPOSITE[DELETE_ALL] operation failed on query: 
                DELETE FROM jos_dbtest
             using args: Array
(
)
 [SQLSTATE[42S02]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Invalid object name 'jos_dbtest'.]
C:\projects\database\vendor\joomla\test\src\TestDatabase.php:218

Additional comments

Possible reference items for solving the issues

Transaction handling is fundamentally broken due to automatic reconnect behaviour in execute() method

Steps to reproduce the issue

It's not that simple to reproduce this error since you have to simulate a database disconnect.

The following code illustrates the problem, it targets MySQL (but all database drivers are affected since every database rolls back transactions automatically when the connection gets lost).

$db = JFactory::getDbo(); // set up connection (Joomla CMS)
$db->setQuery('CREATE TABLE IF NOT EXISTS transactiontest (pk_value INT, somecol INT)');
$db->execute();
try {
    $db->transactionStart();
    $db->setQuery('INSERT INTO transactiontest VALUES (1,1)'); 
    $db->execute(); 
    // to simulate a database disconnect set a brakepoint in the debugger on the next line 
    // and when it get's hit restart your database and then continue debugging when the database is running again
    $db->setQuery('INSERT INTO transactiontest VALUES (2,2)'); 
    // the next line should fail but it does not since the automatic reconnect behaviour inside the execute method
    // will kick in and reconnect sucessfully and just execute the query without triggering an error
    $db->execute(); 
    $db->transactionCommit();
}
catch(RuntimeException $e)
{
    // something failed - rollback
    $db->transactionRollback();
}

$db->setQuery('SELECT * FROM transactiontest');
$rows = $db->loadObjectList();
// excpected result is zero rows, but it returns 1 row with the values 2,2
var_dump($rows); 

Expected result

No rows inserted into the database

Actual result

1 row inserted into the database

System information (as much as possible)

Doesn't matter - since the issue is independent of OS, database or driver that's used.
All platforms are affected.

Additional comments

There are two solutions to the issue.

  1. remove the automatic reconnect code in the execute() method, this is probably problematic since it
    isn't in general a bad idea to do so, intermittend database outages might occur and if one is not inside a transaction the reconnect behaviour is a good thing

  2. only try to reconnect if not inside a transaction - this is actually my proposed solution since
    to implement that just one line needs to be changed in the execute method (of all database classes)

// If an error occurred handle it.
if (!$this->cursor)
{
	// Get the error number and message before we execute any more queries.
	$this->errorNum = $this->getErrorNumber();
	$this->errorMsg = $this->getErrorMessage();

	// Check if the server was disconnected, but only if not inside a transaction
	if ($this->transactionDepth == 0 && !$this->connected())
	{
		try
		{
			// Attempt to reconnect.
			$this->connection = null;
			$this->connect();
		}
...

Suggestion: Incorporate Doctrine DBAL in V2

As per the Framework roadmap "For version 2, we are looking at ... potentially consuming a third party database system."

I don't know what has been discussed so far, but Doctrine DBAL seems like it might be a good candidate for consideration as it is an "Abstraction layer and access library for relational databases. A thin layer on top of PDO with a lot of additional, horizontal functionality."

[2.0] NULL Date check is inconsistent

Steps to reproduce the issue

  1. Install Joomla 4 on MySQL 5.7
  2. Login to the administrator section
  3. The control panel is empty

image

Expected result

The full control panel with modules is shown.

Actual result

The control panel has no modules.

System information (as much as possible)

PHP 7.2.8
MySQL 5.7.22
Client API library version | mysqlnd 5.0.12-dev - 20150407
Joomla 4

Additional comments

The query executed is

SELECT m.id, m.title, m.module, m.position, m.content, m.showtitle, m.params, mm.menuid

  FROM ila8e_modules AS m

  LEFT JOIN ila8e_modules_menu AS mm 
  ON mm.moduleid = m.id

  LEFT JOIN ila8e_extensions AS e 
  ON e.element = m.module 
  AND e.client_id = m.client_id

  WHERE m.published = 1 
  AND e.enabled = 1 
  AND (m.publish_up = '0000-00-00 00:00:00' OR m.publish_up <= '2018-08-01 09:13:47') 
  AND (m.publish_down = '0000-00-00 00:00:00' OR m.publish_down >= '2018-08-01 09:13:47') 
  AND m.access IN (1,1,2,3,6) 
  AND m.client_id = 1 
  AND (mm.menuid = 0 OR mm.menuid <= 0)

  ORDER BY m.position, m.ordering

In the database the NULL date value is 1000-01-01 00:00:00 but as you can see the query uses 0000-00-00 00:00:00.

In this file ( https://github.com/joomla-framework/database/blob/2.0-dev/src/Mysqli/MysqliDriver.php#L930 ) the getNullDate() checks for the NO_ZERO_DATE SQL Mode. Running the query SELECT @@SESSION.sql_mode; through Joomla/PHP returns

ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

however, running the same query on Sequel Pro or the MySQL CLI it returns

ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

Notice that the NO_ZERO_DATE is there when checking directly on MySQL or using a different client but not when using Joomla/PHP.

I believe we need an additional check here.

Statement nullified after loading results

Steps to reproduce the issue

Run a prepared statement in a loop.

$ids = [1, 2, 3];

$query
	->select('*')
	->from('#__table')
	->where('id = :id')
	->bind(':id', $id, ParameterType::INTEGER);
$db->setQuery($query);

foreach ($ids as $id)
{
	$db->loadObject();
}

Expected result

Rows with specified IDs loaded.

Actual result

Call to a member function bindParam() on null

System information (as much as possible)

PDO and MySQLi.

Additional comments

I'm again not sure whether this is expected or not. One can just move $db->setQuery() inside the loop to workaround this. However, this is not required when performing queries that are executed with $db->execute() (e.g. insert/update). To my (limited) understanding, given the nature of prepared statements, DatabaseDriver::freeResult() should nullify only the result set and not the entire statement.

Error handling of PDO driver broken

Steps to reproduce the issue

  1. Edit your postgres install SQL and set the sequence ID of the assets table to something lower than the already used keys.
  2. Install Joomla with Postgres (PDO)
  3. Create a menu and save it.

Expected result

An error screen maybe with a stack trace and an error message that gives any hint what went wrong.

Actual result

You get a white screen with the text Recursion trying to check if connected.

System information (as much as possible)

Checkout from 4.0-dev

Additional comments

The issue seems to be with Joomla\Database\Pdo\PdoDriver::connected(). That method should check if the connection is alive and for that runs a query, which in turn calls execute(), which again calls connected() which then runs into the recursion condition.

Deprecation warning for interfaces

We deprecate PreparableInterface and LimitableInterface in 2.0 and trigger 2 E_USER_DEPRECATED on each request.

From performance pov it's not optimal especially as this call is prefixed with @ (Warning suppression). And the CMS will use this several years and can't switch to 3.0 until J5.0.

Can we remove the interface from QueryInterface and merge the function into QueryInterface?

I ask because I want to create a trait to the PreparableInterface but I'm not sure if this is useful if it get removed in 3.0.

Tests causing build errors due to testing with JTEST_DATABASE_PGSQL_DSN

The following tests in JTEST_DATABASE_PGSQL_DSN are causing php unit to fail to complete testing;
due to these tests JTEST_DATABASE_PGSQL_DSN has been commented out in phpunit.travis.xml.

  • DriverPostgresqlTest::testExecutePreparedStatement
  • DriverPostgresqlTest::testRenameTable
  • DriverPostgresqlTest::testTransactionStart
  • DriverPostgresqlTest::testTransactionCommit
  • DriverPostgresqlTest::testInsertObject
  • DriverPostgresqlTest::testInsertid
  • DriverPgsqlTest::testRenameTable

These tests could be excluded from the test suit until they are fixed so more tests are completed.

For example, once the tests are excluded the following IteratorPgsqlTest failures are reported

There were 5 failures:
1) Joomla\Database\Tests\IteratorPgsqlTest::testForEach with data set #0 ('title', '#__dbtest', null, 'stdClass', 0, 0, array(stdClass Object (...), stdClass Object (...), stdClass Object (...), stdClass Object (...)), null)
136
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => stdClass Object (
-        'title' => 'Testing'
+        'title' => 'testTitle'
     )
     1 => stdClass Object (
-        'title' => 'Testing2'
+        'title' => 'Testing'
     )
     2 => stdClass Object (
-        'title' => 'Testing3'
+        'title' => 'Testing2'
     )
     3 => stdClass Object (
-        'title' => 'Testing4'
+        'title' => 'Testing3'
     )
+    4 => stdClass Object (...)
 )
/database/Tests/IteratorPgsqlTest.php:137

2) Joomla\Database\Tests\IteratorPgsqlTest::testForEach with data set #1 ('title', '#__dbtest', null, 'stdClass', 2, 0, array(stdClass Object (...), stdClass Object (...)), null)
136
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => stdClass Object (
-        'title' => 'Testing'
+        'title' => 'testTitle'
     )
     1 => stdClass Object (
-        'title' => 'Testing2'
+        'title' => 'Testing'
     )
 )
/database/Tests/IteratorPgsqlTest.php:137

3) Joomla\Database\Tests\IteratorPgsqlTest::testForEach with data set #2 ('title', '#__dbtest', null, 'stdClass', 20, 2, array(stdClass Object (...), stdClass Object (...)), null)
136
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => stdClass Object (
-        'title' => 'Testing3'
+        'title' => 'Testing2'
     )
     1 => stdClass Object (
-        'title' => 'Testing4'
+        'title' => 'Testing3'
     )
+    2 => stdClass Object (...)
 )
/database/Tests/IteratorPgsqlTest.php:137

4) Joomla\Database\Tests\IteratorPgsqlTest::testForEach with data set #3 ('title, id', '#__dbtest', 'title', 'stdClass', 0, 0, array(stdClass Object (...), stdClass Object (...), stdClass Object (...), stdClass Object (...)), null)
136
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     'Testing' => stdClass Object (...)
     'Testing2' => stdClass Object (...)
     'Testing3' => stdClass Object (...)
     'Testing4' => stdClass Object (...)
+    'testTitle' => stdClass Object (...)
 )
/database/Tests/IteratorPgsqlTest.php:137

5) Joomla\Database\Tests\IteratorPgsqlTest::testCount
153
Failed asserting that 5 matches expected 4.
/database/Tests/IteratorPgsqlTest.php:154

Union once again

Steps to reproduce the issue

Query with Union does not work at ll.

My example:

Expected result

Should work.

Actual result

System information (as much as possible)

Joomla 4.0-dev

Additional comments

Is something wrong with do it in the same way as in Joomla3?

I propose a fix:

diff --git a/src/DatabaseQuery.php b/src/DatabaseQuery.php
index 7ee91d0..e912e01 100644
--- a/src/DatabaseQuery.php
+++ b/src/DatabaseQuery.php
@@ -267,6 +267,16 @@ abstract class DatabaseQuery implements QueryInterface
                                        {
                                                $query .= (string) $this->having;
                                        }
+
+                                       if ($this->union)
+                                       {
+                                               $query .= (string) $this->union;
+                                       }
+
+                                       if ($this->unionAll)
+                                       {
+                                               $query .= (string) $this->unionAll;
+                                       }
                                }
 
                                if ($this->order)

Ping @mbabker

Query implementing PreparableInterface throws Exception

I'm using the MysqliDriver::lockTable() method.

With the commits of recent days (nearly 382f9cb), this method throws an exception with this message : "This command is not supported in the prepared statement protocol yet.".

I'm using MariaDB server (I don't know if it is important).
Imo, the LOCK TABLES statement cannot be prepared.

add Oracle CI testing

Steps to reproduce the issue

no continuous integration testing is currently performed on some database platforms like Oracle, etc

Expected result

Test against all our database platforms like Oracle, etc

Actual result

no continuous integration testing is currently performed of the Oracle driver

Additional comments

Apparently, someone has a workaround for testing Oracle DB on travis ci as well https://github.com/cbandy/travis-oracle

an Oracle account is required and the Oracle username and password has to be in the build environment variables either as hidden repository settings or encrypted variables

`mysqli_stmt::get_result()` is not available on HHVM

Steps to reproduce the issue

Review Travis HHVM tests, mysqli_stmt::get_result() is not available on HHVM

Expected result

no error

Actual result

Fatal error: Call to undefined method mysqli_stmt::get_result() in /src/Mysqli/MysqliDriver.php on line 635

System information (as much as possible)

HHVM 3.15.3

Additional comments

HHVM won't fix this but would take a PR to add the functionality
facebook/hhvm#2368

As such we have 3 options

  1. Explicitly drop HHVM support
  2. Submit a PR to HHVM to add the functionality
  3. Rewrite the portion of the driver to iterate/loop over the results and return all of the results.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.