mongodb-labs / python-bsonjs Goto Github PK
View Code? Open in Web Editor NEWA fast BSON to MongoDB Extended JSON converter for Python - This Repository is NOT a supported MongoDB product
License: Apache License 2.0
A fast BSON to MongoDB Extended JSON converter for Python - This Repository is NOT a supported MongoDB product
License: Apache License 2.0
Windows Python 3.8 compile warnings in libbson 1.17.0:
$ /cygdrive/c/python/Python38/python setup.py build_ext -i
running build_ext
building 'bsonjs' extension
creating build
creating build\temp.win-amd64-3.8
creating build\temp.win-amd64-3.8\Release
creating build\temp.win-amd64-3.8\Release\src
creating build\temp.win-amd64-3.8\Release\src\bson
creating build\temp.win-amd64-3.8\Release\src\common
creating build\temp.win-amd64-3.8\Release\src\jsonsl
...
C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.25.28610\bin\HostX64\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DBSON_COMPILATION=1 -Isrc -Isrc/bson -Isrc/jsonsl -Isrc/common -IC:\python\Python38\include -IC:\python\Python38\includ
e "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.25.28610\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.25.28610\ATLMFC\Include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.
18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\
4.7.2\include\um" /Tcsrc\bson\bson-atomic.c /Fobuild\temp.win-amd64-3.8\Release\src\bson\bson-atomic.obj
bson-atomic.c
src\bson\bson-atomic.c(93): warning C4133: 'function': incompatible types - from 'volatile int64_t *' to 'volatile LONG *'
src\bson\bson-atomic.c(93): warning C4244: 'function': conversion from 'int64_t' to 'LONG', possible loss of data
src\bson\bson-context.c(290): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
src\bson\bson-decimal128.c(176): warning C4996: 'strcpy': This function or variable may be unsafe. Consider using strcpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
src\bson\bson-decimal128.c(180): warning C4996: 'strcpy': This function or variable may be unsafe. Consider using strcpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
src\bson\bson-decimal128.c(629): warning C4267: '-=': conversion from 'size_t' to 'int32_t', possible loss of data
src\bson\bson-iter.c(114): warning C4267: '=': conversion from 'size_t' to 'uint32_t', possible loss of data
src\bson\bson-json.c(617): warning C4146: unary minus operator applied to unsigned type, result still unsigned
src\bson\bson-json.c(1069): warning C4267: 'function': conversion from 'size_t' to 'int', possible loss of data
src\bson\bson-json.c(2090): warning C4018: '<': signed/unsigned mismatch
src\bson\bson-json.c(2091): warning C4018: '<': signed/unsigned mismatch
src\bson\bson-timegm.c(488): warning C4244: '+=': conversion from 'const int64_t' to 'int_fast32_t', possible loss of data
src\jsonsl\jsonsl.c(921): warning C4996: 'strcpy': This function or variable may be unsafe. Consider using strcpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
src\jsonsl\jsonsl.c(959): warning C4996: 'strcpy': This function or variable may be unsafe. Consider using strcpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
python-bsonjs currently relies on the libbson library being installed on the system. Vendoring libbson will avoid this requirement and make installing as simple as
$ pip install python-bsonjs
Libbson 1.6 included some major improvements to it's bson to json encoding:
Currently on master the test_binary test fails with the following error:
FAIL: test_binary (test.test_bsonjs.TestBsonjs)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/julius/python-bsonjs/test/test_bsonjs.py", line 195, in test_binary
self.round_trip(bin_type_dict)
File "/Users/julius/python-bsonjs/test/test_bsonjs.py", line 79, in round_trip
self.assertEqual(doc, json_util.loads(
AssertionError: {'bin': Binary(b'\x00\x01\x02\x03\x04', 0)} != {'bin': b'\x00\x01\x02\x03\x04'}
- {'bin': Binary(b'\x00\x01\x02\x03\x04', 0)}
? ------- ----
+ {'bin': b'\x00\x01\x02\x03\x04'}
For example, I am inserting 184.8k documents after reading records from an SQL database. The number of documents shown on compassCommunity after insertion is 184.8k but once I use group aggregation to count number of documents , the result is 100k. The aggregation query I am using is - [ { '$group': { '_id': None, 'count': { '$sum': 1 } } } ]
We should test on Evergreen which already has all the pythons and compilers we need to test on Windows. Alternatively we could try out Travis' experimental Windows support, Azure Pipelines, AppVeyor, or GitHub Actions.
Add operating systems bsonjs is known to work on:
"Operating System :: MacOS :: MacOS X",
"Operating System :: Microsoft :: Windows",
"Operating System :: POSIX",
Once libbson 1.7.0 is released with MongoDB Extended JSON 2.0 support we can add support here as well. See https://jira.mongodb.org/browse/CDRIVER-2128.
Libbson 1.4.0 fixes a couple issues regarding BSON to MongoDB extended JSON conversion including:
We should drop support for Python 2.7, 3.4, and 3.5 which are all EOL.
Failed to build python-bsonjs
DEPRECATION: Could not build wheels for python-bsonjs which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
Installing collected packages: python-bsonjs
Running setup.py install for python-bsonjs ... error
ERROR: Command errored out with exit status 1:
command: 'c:\python\toolusagelogger\_env\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\bam5wi\\AppData\\Local\\Temp\\pip-install-dutuekdi\\python-bsonjs\\setup.py'"'"'; __file__='"'"'C:\\Users\\bam5wi\\AppData\\Local\\Temp\\pip-install-dutuekdi\\python-bsonjs\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\bam5wi\AppData\Local\Temp\pip-record-dbaa8bjo\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\python\toolusagelogger\_env\include\site\python3.8\python-bsonjs'
cwd: C:\Users\bam5wi\AppData\Local\Temp\pip-install-dutuekdi\python-bsonjs\
Complete output (75 lines):
running install
running build
running build_ext
building 'bsonjs' extension
creating build
creating build\temp.win-amd64-3.8
creating build\temp.win-amd64-3.8\Release
creating build\temp.win-amd64-3.8\Release\src
creating build\temp.win-amd64-3.8\Release\libbson
creating build\temp.win-amd64-3.8\Release\libbson\src
creating build\temp.win-amd64-3.8\Release\libbson\src\bson
creating build\temp.win-amd64-3.8\Release\libbson\src\jsonsl
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\Bin\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DBSON_COMPILATION=1 -Isrc -Isrc/bson -Ilibbson/src -Ilibbson/src/jsonsl -Ilibbson/src/bson -Ic:\python\toolusagelogger\_env\include -IC:\Python\_envs\Python38\include -IC:\Python\_envs\Python38\include "-IC:\Program Files\Microsoft SDKs\Windows\v7.1\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include" -IC:\Python\ToolUsageLogger\_env\Include -IC:\Python\ToolUsageLogger\_env\Include\cairo "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.10150.0\ucrt" /Tcsrc/bsonjs.c /Fobuild\temp.win-amd64-3.8\Release\src/bsonjs.obj
bsonjs.c
c:\python\_envs\python38\include\object.h(436) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\object.h(437) : error C2085: '_Py_NewReference' : not in formal parameter list
c:\python\_envs\python38\include\object.h(437) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\object.h(446) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\object.h(447) : error C2085: '_Py_ForgetReference' : not in formal parameter list
c:\python\_envs\python38\include\object.h(447) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\object.h(456) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\object.h(457) : error C2085: '_Py_INCREF' : not in formal parameter list
c:\python\_envs\python38\include\object.h(457) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\object.h(464) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\object.h(466) : error C2085: '_Py_DECREF' : not in formal parameter list
c:\python\_envs\python38\include\object.h(466) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\object.h(529) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\object.h(530) : error C2085: '_Py_XINCREF' : not in formal parameter list
c:\python\_envs\python38\include\object.h(530) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\object.h(538) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\object.h(539) : error C2085: '_Py_XDECREF' : not in formal parameter list
c:\python\_envs\python38\include\object.h(539) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\cpython/object.h(339) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\cpython/object.h(340) : error C2085: '_Py_Dealloc_inline' : not in formal parameter list
c:\python\_envs\python38\include\cpython/object.h(340) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\objimpl.h(136) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\objimpl.h(138) : error C2085: '_PyObject_INIT' : not in formal parameter list
c:\python\_envs\python38\include\objimpl.h(138) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\objimpl.h(151) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\objimpl.h(153) : error C2085: '_PyObject_INIT_VAR' : not in formal parameter list
c:\python\_envs\python38\include\objimpl.h(153) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\cpython/abstract.h(75) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\cpython/abstract.h(76) : error C2085: 'PyVectorcall_NARGS' : not in formal parameter list
c:\python\_envs\python38\include\cpython/abstract.h(76) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\cpython/abstract.h(81) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\cpython/abstract.h(82) : error C2085: '_PyVectorcall_Function' : not in formal parameter list
c:\python\_envs\python38\include\cpython/abstract.h(82) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\cpython/abstract.h(114) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\cpython/abstract.h(117) : error C2085: '_PyObject_Vectorcall' : not in formal parameter list
c:\python\_envs\python38\include\cpython/abstract.h(117) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\cpython/abstract.h(144) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\cpython/abstract.h(146) : error C2085: '_PyObject_FastCall' : not in formal parameter list
c:\python\_envs\python38\include\cpython/abstract.h(146) : error C2143: syntax error : missing ';' before '{'
c:\python\_envs\python38\include\cpython/abstract.h(151) : error C2054: expected '(' to follow 'inline'
c:\python\_envs\python38\include\cpython/abstract.h(152) : error C2085: '_PyObject_CallNoArg' : not in formal parameter list
c:\python\_envs\python38\include\cpython/abstract.h(152) : error C2143: syntax error : missing ';' before '{'
c:\program files\microsoft sdks\windows\v7.1\include\sal_supp.h(57) : warning C4005: '__useHeader' : macro redefinition
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\sal.h(2886) : see previous definition of '__useHeader'
c:\program files\microsoft sdks\windows\v7.1\include\specstrings_supp.h(77) : warning C4005: '__on_failure' : macro redefinition
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\sal.h(2896) : see previous definition of '__on_failure'
c:\users\bam5wi\appdata\local\temp\pip-install-dutuekdi\python-bsonjs\libbson\src\bson\bson-compat.h(110) : warning C4005: 'PRIi32' : macro redefinition
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\inttypes.h(78) : see previous definition of 'PRIi32'
c:\users\bam5wi\appdata\local\temp\pip-install-dutuekdi\python-bsonjs\libbson\src\bson\bson-compat.h(111) : warning C4005: 'PRId32' : macro redefinition
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\inttypes.h(68) : see previous definition of 'PRId32'
c:\users\bam5wi\appdata\local\temp\pip-install-dutuekdi\python-bsonjs\libbson\src\bson\bson-compat.h(112) : warning C4005: 'PRIu32' : macro redefinition
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\inttypes.h(98) : see previous definition of 'PRIu32'
c:\users\bam5wi\appdata\local\temp\pip-install-dutuekdi\python-bsonjs\libbson\src\bson\bson-compat.h(113) : warning C4005: 'PRIi64' : macro redefinition
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\inttypes.h(132) : see previous definition of 'PRIi64'
c:\users\bam5wi\appdata\local\temp\pip-install-dutuekdi\python-bsonjs\libbson\src\bson\bson-compat.h(114) : warning C4005: 'PRId64' : macro redefinition
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\inttypes.h(126) : see previous definition of 'PRId64'
c:\users\bam5wi\appdata\local\temp\pip-install-dutuekdi\python-bsonjs\libbson\src\bson\bson-compat.h(115) : warning C4005: 'PRIu64' : macro redefinition
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\inttypes.h(144) : see previous definition of 'PRIu64'
src/bsonjs.c(102) : warning C4013: '_Py_DECREF' undefined; assuming extern returning int
src/bsonjs.c(107) : warning C4013: '_Py_INCREF' undefined; assuming extern returning int
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 12.0\\VC\\Bin\\x86_amd64\\cl.exe' failed with exit status 2
The latest wheels available on PyPI are for Python 3.6. It would be helpful to have prebuilt wheels available for more recent Python versions.
Those are both long dead and gone and PyMongo no longer supports either of them.
We should fix these compile warnings:
/bsonjs.c -o build/temp.linux-x86_64-3.10/src/bsonjs.o
src/bsonjs.c:234:14: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
{"dump", dump, METH_VARARGS | METH_KEYWORDS, dump__doc__},
^~~~
src/bsonjs.c:234:14: note: (near initialization for ‘BsonjsClientMethods[0].ml_meth’)
src/bsonjs.c:235:15: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
{"dumps", dumps, METH_VARARGS | METH_KEYWORDS, dumps__doc__},
^~~~~
Libbson 1.6.2 directly includes stdint.h which is not provided by the version of visual studio used by Python 2.6 and 2.7 on Windows.
PS D:\Git\python-bsonjs2> D:\Python26\python.exe .\setup.py build_ext
running build_ext
building 'bsonjs' extension
creating build
creating build\temp.win-amd64-2.6
creating build\temp.win-amd64-2.6\Release
creating build\temp.win-amd64-2.6\Release\src
creating build\temp.win-amd64-2.6\Release\libbson
creating build\temp.win-amd64-2.6\Release\libbson\src
creating build\temp.win-amd64-2.6\Release\libbson\src\bson
creating build\temp.win-amd64-2.6\Release\libbson\src\jsonsl
C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD
/W3 /GS- /DNDEBUG -DBSON_COMPILATION=1 -Isrc -Isrc/bson -Ilibbson/src -Ilibbson/src/jsonsl -Ilibbson/src/bson -ID:\Pytho
n26\include -ID:\Python26\PC /Tcsrc/bsonjs.c /Fobuild\temp.win-amd64-2.6\Release\src/bsonjs.obj
bsonjs.c
C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD
/W3 /GS- /DNDEBUG -DBSON_COMPILATION=1 -Isrc -Isrc/bson -Ilibbson/src -Ilibbson/src/jsonsl -Ilibbson/src/bson -ID:\Pytho
n26\include -ID:\Python26\PC /Tclibbson/src\bson\bcon.c /Fobuild\temp.win-amd64-2.6\Release\libbson/src\bson\bcon.obj
bcon.c
C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD
/W3 /GS- /DNDEBUG -DBSON_COMPILATION=1 -Isrc -Isrc/bson -Ilibbson/src -Ilibbson/src/jsonsl -Ilibbson/src/bson -ID:\Pytho
n26\include -ID:\Python26\PC /Tclibbson/src\bson\bson-atomic.c /Fobuild\temp.win-amd64-2.6\Release\libbson/src\bson\bson
-atomic.obj
bson-atomic.c
C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD
/W3 /GS- /DNDEBUG -DBSON_COMPILATION=1 -Isrc -Isrc/bson -Ilibbson/src -Ilibbson/src/jsonsl -Ilibbson/src/bson -ID:\Pytho
n26\include -ID:\Python26\PC /Tclibbson/src\bson\bson-clock.c /Fobuild\temp.win-amd64-2.6\Release\libbson/src\bson\bson-
clock.obj
bson-clock.c
C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD
/W3 /GS- /DNDEBUG -DBSON_COMPILATION=1 -Isrc -Isrc/bson -Ilibbson/src -Ilibbson/src/jsonsl -Ilibbson/src/bson -ID:\Pytho
n26\include -ID:\Python26\PC /Tclibbson/src\bson\bson-context.c /Fobuild\temp.win-amd64-2.6\Release\libbson/src\bson\bso
n-context.obj
bson-context.c
C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD
/W3 /GS- /DNDEBUG -DBSON_COMPILATION=1 -Isrc -Isrc/bson -Ilibbson/src -Ilibbson/src/jsonsl -Ilibbson/src/bson -ID:\Pytho
n26\include -ID:\Python26\PC /Tclibbson/src\bson\bson-decimal128.c /Fobuild\temp.win-amd64-2.6\Release\libbson/src\bson\
bson-decimal128.obj
bson-decimal128.c
libbson/src\bson\bson-decimal128.c(19) : fatal error C1083: Cannot open include file: 'stdint.h': No such file or direct
ory
error: command '"C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe"'
failed with exit status 2
Libbson no longer builds on Windows with Python 2.7:
$ /cygdrive/c/python/Python27/python setup.py test
running test
running egg_info
creating python_bsonjs.egg-info
writing python_bsonjs.egg-info\PKG-INFO
writing top-level names to python_bsonjs.egg-info\top_level.txt
writing dependency_links to python_bsonjs.egg-info\dependency_links.txt
writing manifest file 'python_bsonjs.egg-info\SOURCES.txt'
reading manifest file 'python_bsonjs.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.h' under directory 'libbson'
writing manifest file 'python_bsonjs.egg-info\SOURCES.txt'
running build_ext
building 'bsonjs' extension
creating build
creating build\temp.win-amd64-2.7
creating build\temp.win-amd64-2.7\Release
creating build\temp.win-amd64-2.7\Release\src
creating build\temp.win-amd64-2.7\Release\src\bson
creating build\temp.win-amd64-2.7\Release\src\common
creating build\temp.win-amd64-2.7\Release\src\jsonsl
C:\Program Files (x86)\Common Files\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -DBS
ON_COMPILATION=1 -Isrc -Isrc/bson -Isrc/jsonsl -Isrc/common -IC:\python\Python27\include -IC:\python\Python27\PC /Tcsrc/bsonjs.c /Fob
uild\temp.win-amd64-2.7\Release\src/bsonjs.obj
bsonjs.c
c:\cygwin\home\administrator\python-bsonjs\src\bson\bson-compat.h(80) : fatal error C1083: Cannot open include file: 'stdint.h': No s
uch file or directory
error: command 'C:\\Program Files (x86)\\Common Files\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\cl.exe' failed with exi
t status 2
This was changed in libbson 1.10: https://jira.mongodb.org/browse/CDRIVER-2473 when the minimum version of Visual Studio was raised to VS2010. The Microsoft Visual C++ Compiler for Python 2.7 uses VS2008 and does not include stdint.h.
Our options are to:
Dear Shane,
thanks a stack for conceiving and maintaining the excellent python-bsonjs
package.
On a CI run of ours 1, we have been able to spot this error:
ERROR: Could not build wheels for python-bsonjs, which is required to install pyproject.toml-based projects
This is probably to be expected, because 2 shows there are no wheel packages available for more recent versions of Python, and the environment where python-bsonjs
is being installed lacks the gcc
compiler package.
Do you see any chance to publish wheel packages to PyPI, to improve this situation?
With kind regards,
Andreas.
The current latest is 1.14.0: https://github.com/mongodb/mongo-c-driver/tree/1.14.0/src/libbson/
Also note that libbson is now part of the mongo-c-driver repo.
Hi Shane,
We have a requirment to process all audit bson file of Mongo database and store it at a centralized location for reporting. In our current process, we scan all the audit bson file convert it to json and send the json file to be persisted at centralized location via REST API call. Following is a snippet of code...
**from bson.json_util import loads, dumps, DEFAULT_JSON_OPTIONS
from bson import decode_all
if not self.util.isFileExists(auditFile):
return self.util.buildResponse(self.Globals.unsuccess, f"file {auditFile} is missing ")
myAuditFileSize = self.util.getFileSizeBytes(auditFile) / (1024*1024)
if myAuditFileSize > self.BSON_FILE_SIZE_LIMIT_MB:
print(f"Audit bson file '{auditFile}' size is larger than {self.BSON_FILE_SIZE_LIMIT_MB}")
return self.util.buildResponse(self.Globals.unsuccess, f"Audit bson file '{auditFile}' size is larger than {self.BSON_FILE_SIZE_LIMIT_MB}MB ")
try:
if self.util.getFileExtn(auditFile).lower() == "json":
myMongoAuditData = self.util.readJsonFile(auditFile)
else:
with open(auditFile, 'rb') as file:
myMongoAuditData = decode_all(file.read())
return myMongoAuditData**
We are facing issue on processing larger bson file thus restricting the size of audit bson file which will be processed. I need your help to use "bsonjs" module to process the audit bson file to generate the json file (will be better to generate smaller json file).
Pls assist.
Thanks,
Anil Kumar
if you do the following:
bsonjs.loads(json.dumps([]))
The result is {}
This caused some minor issues in a program I'm working on. Since json is able to load lists I think there is no reason bson shouldn't experience similar behavior.
i was experimenting with bsonjs. insert_one works fine, however when I tried insert_many, I got the following error msg: "File "C:\ProgramData\Anaconda2\lib\site-packages\pymongo\common.py", line 453, in validate_is_document_type
"collections.MutableMapping" % (option,))
TypeError: document must be an instance of dict, bson.son.SON, bson.raw_bson.RawBSONDocument, or a type that inherits from collections.MutableMapping".
I casted "rawBS1 = RawBSONDocument(bson_bytes)" just the line before, and worked fine with insert_one.
I've been going around in circles a bit trying to understand if this library can be used to speed up decoding an aggregation query.
Or whether after recent pymongo updates its needed at all.
documents = []
cursor = collection.aggregate_raw_batches(
pipeline=aggregation_query,
)
while True:
try:
documents.extend([x for x in decode_all(cursor.next())])
except StopIteration:
break
How would I use bsonjs.dumps instead?
Any chance of getting a new release with the updated libbson that fixes known issues? I've found this library to be necessary for large ELT processes using pymongo and would love to see it continue.
Hi,
First of all, it looks like an extremely interesting library that could really improve some of the performance issues in pymongo with larger documents! So this might be maybe more of a question maybe than an issue, but this might help other people too.
So normally if I put something into gridfs, i use bson.json_util.dumps to do a conversion:
dict -> json string -> bson binary.
From what I read on pymongo, bsonjs is a stand-in for it (with less options but much better performance). However as far as i can see, bsonjs doesn't do exactly the same as json_util but only the
json string -> bson binary
So I need to construct the json string first myself, for which i can use bson.json_util or another library like ujson:
%timeit a = bson.json_util.dumps(nested_dict)
1 loop, best of 3: 255 ms per loop
%timeit a = ujson.dumps(nested_dict)
10 loops, best of 3: 24.4 ms per loop
So this makes it about 10x faster, which is great. But I cannot always use ujson (because of ObjectIds) and bsonjs itself falls over encodings in strings which i cannot control with ujson.
Question 1: Unless I have a quick way to generate the json string, there is no real advantage to it? So i need to work with these constraints right?
Question 2: Is there any way to also write raw documents to GridFS like you documented for the document_class? Or does that happen automatically if the used db is in Raw BSON mode?
Question 3: How does it work for (non-GridFS, normal collections) updates? do i need to encode the {'_id': ObjectId('123456789')}, {'$set': {'foo': 'bar'}} as two raw bson documents?
Thanks!
Error:
PS D:\Git\python-bsonjs> python setup.py build_ext
running build_ext
building 'bsonjs' extension
C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD
/W3 /GS- /DNDEBUG -DBSONJS_VERSION="0.1.0" -DBSON_COMPILATION=1 -Isrc -Isrc/bson -Ilibbson/src -Ilibbson/src/yajl -Ilibb
son/src/bson -IC:\Python27\include -IC:\Python27\PC /Tcsrc/bsonjs.c /Fobuild\temp.win-amd64-2.7\Release\src/bsonjs.obj
bsonjs.c
src/bsonjs.c(258) : error C2440: 'function' : cannot convert from 'double' to 'const char *'
src/bsonjs.c(258) : warning C4024: 'PyString_FromString' : different types for formal and actual parameter 1
src/bsonjs.c(258) : error C2143: syntax error : missing ')' before 'constant'
src/bsonjs.c(259) : error C2181: illegal else without matching if
src/bsonjs.c(259) : error C2059: syntax error : 'while'
src/bsonjs.c(260) : error C2059: syntax error : 'return'
src/bsonjs.c(261) : error C2059: syntax error : '}'
error: command 'C:\\Users\\admin\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd6
4\\cl.exe' failed with exit status 2
Cause:
The double quotes around BSONJS_VERSION in setup.py are lost resulting in the macro:
#define BSONJS_VERSION 0.1.0
instead of:
#define BSONJS_VERSION "0.1.0"
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.