opendap / bes Goto Github PK
View Code? Open in Web Editor NEWThe BES framework, which forms the basis for the Hyrax server
License: GNU Lesser General Public License v3.0
The BES framework, which forms the basis for the Hyrax server
License: GNU Lesser General Public License v3.0
Our 1.13.1 server in operation had the BES cache filled in the last couple days. Based on the bes.conf spec below I wonder if the server should have started purging cached files when it reached 500MB? The cache is a 100GB area.
BES.UncompressCache.dir=/tmpdata/bes
BES.UncompressCache.prefix=bes_cache
BES.UncompressCache.size=500
BES.Uncompress.Retry=2000
BES.Uncompress.NumTries=10
After fixing libdap, I get further compilation errors:
make[6]: Entering directory '/home/rjones/d/fedora/bes/rawhide/bes-3.21.0-46/modules/hdf5_handler/gctp/src'
gcc -DHAVE_CONFIG_H -I. -I../../../.. -I./../include/ -I/usr/include/cfitsio -I/usr/include/tirpc -Wno-vla -I/include -fpic -Wno-implicit-function-declaration -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -c -o gctp.o gctp.c
gctp.c: warning: -Wno-implicit-function-declaration detected - is this intentional ?
gctp.c: In function ‘gctp’:
gctp.c:257:81: error: passing argument 8 of ‘hinv_init’ from incompatible pointer type [-Wincompatible-pointer-types]
257 | hinv_init((int)*insys,(int)*inzone,dummy,(int)*indatum,fn27,fn83,&iflgval,(int *)hinv_trans);
| ^~~~~~~~~~~~~~~~~
| |
| int *
In file included from ./../include/cproj_prototypes.h:32,
from ./../include/cproj.h:33,
from gctp.c:33:
./../include/gctp_prototypes.h:288:108: note: expected ‘int (**)(double, double, double *, double *)’ but argument is of type ‘int *’
288 | int hinv_init(int insys, int inzone, double *inparm, int indatum, char *fn27, char *fn83, int *iflg, int (*hinv_trans[])(double, double, double*, double*));
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gctp.c:260:81: error: passing argument 8 of ‘hinv_init’ from incompatible pointer type [-Wincompatible-pointer-types]
260 | hinv_init((int)*insys,(int)*inzone,inparm,(int)*indatum,fn27,fn83,&iflgval,(int *)hinv_trans);
| ^~~~~~~~~~~~~~~~~
| |
| int *
./../include/gctp_prototypes.h:288:108: note: expected ‘int (**)(double, double, double *, double *)’ but argument is of type ‘int *’
288 | int hinv_init(int insys, int inzone, double *inparm, int indatum, char *fn27, char *fn83, int *iflg, int (*hinv_trans[])(double, double, double*, double*));
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gctp.c:319:84: error: passing argument 8 of ‘hfor_init’ from incompatible pointer type [-Wincompatible-pointer-types]
319 | hfor_init((int)*outsys,(int)*outzone,dummy,(int)*outdatum,fn27,fn83,&iflgval,(int *)hfor_trans);
| ^~~~~~~~~~~~~~~~~
| |
| int *
./../include/gctp_prototypes.h:284:112: note: expected ‘int (**)(double, double, double *, double *)’ but argument is of type ‘int *’
284 | int hfor_init(int outsys, int outzone, double *outparm, int outdatum, char *fn27, char *fn83, int *iflg, int (*hfor_trans[])(double, double, double *, double *));
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gctp.c:322:85: error: passing argument 8 of ‘hfor_init’ from incompatible pointer type [-Wincompatible-pointer-types]
322 | hfor_init((int)*outsys,(int)*outzone,outparm,(int)*outdatum,fn27,fn83,&iflgval,(int *)hfor_trans);
| ^~~~~~~~~~~~~~~~~
| |
| int *
./../include/gctp_prototypes.h:284:112: note: expected ‘int (**)(double, double, double *, double *)’ but argument is of type ‘int *’
284 | int hfor_init(int outsys, int outzone, double *outparm, int outdatum, char *fn27, char *fn83, int *iflg, int (*hfor_trans[])(double, double, double *, double *));
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
make[6]: *** [Makefile:665: gctp.o] Error 1
This is with:
libdap-3.21.0.27-1.fc41.x86_64
bes-3.21.0-46
Fix.
I just thought I'd try this instead of Jira...
The fileout_covjson can/does return invalid JSON in OPeNDAP Hyrax (1.16.0). Specifically, if we take SPURS_wg_ASL2.nc.covjson as an example
-nan
is present in the domain/axes/x/values
which is not permitted in JSON syntaxdomain/axes
object{
"type": "Coverage",
"domain": {
"type" : "Domain",
"domainType": "Grid",
"axes": {
"x": {
"values": [...]
},
"y": {
"values": [...]
},
"t": {
"values": ["2018-01-01T00:12:20Z"]
}
} <<<<<<<<<<<<<<<<<< COMMA MISSING HERE
"referencing": [{
"coordinates": ["t"],
"system": {
"type": "TemporalRS",
"calendar": "Gregorian"
}
},
...
"parameters": {
"lon_std": {
"type": "Parameter",
"description": {
"en": "Standard deviation for longitude", <<<<<<<< UNNECESSARY COMMA
},
"unit": {
"label": {
"en": "degrees_east"
}
},
...
parameters
block...
},
"observedProperty": {
"label": {
"en": "Number of surface current direction measurements used"
}
}
}
}, <<<<<<<<<<< UNNECESSARY COMMA
},
"ranges": {
"lon_std": {
"type": "NdArray",
...
"cur_dir_ndata": {
"type": "NdArray",
"dataType": "float",
"axisNames": ["t", "y", "x"],
"shape": [1],
"values": [...]
}
}
}] <<<<<<<<<<< CLOSING SQUARE BRACKET AND ADDITIONAL CURLY BRACE RESULT IN INVALID JSON SYNTAX
}
There appear to both other issues as well but I will address them in a separate ticket.
I want to put NetCDF files to hadoop, how to read it by BES? How to Configure bes or modify it?
Thank U~
We have option:
AC_ARG_ENABLE([developer],
[AS_HELP_STRING([--enable-developer],
[Developer mode: Won't override CXXFLAGS on the command line])]
)
However, in spite of that fine promise, it does in fact override CXXFLAGS on the command line. ;-)
In our test server of Hyrax 1.15.1, if I request a covjson file for a granule, most returns as 'CoverageCollection'. Some of these are globally gridded data with multiple observations (arrays) sharing a single pair of latitude and longitude coordinates. For example
Should the granule be interpreted as a 'Coverage' or a 'CoverageCollection'?
Instead of generating a new DMR++ file with the -o
option, add the ability to run tests (-T, -I, -F) against an already-generated DMR++ file.
Making all in xmlcommand
make[2]: Entering directory '/home/ruby/rpmbuild/SOURCES/bes-3.21.0/xmlcommand'
Making all in .
make[3]: Entering directory '/home/ruby/rpmbuild/SOURCES/bes-3.21.0/xmlcommand'
depbase=`echo BESXMLInterface.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLInterface.lo -MD -MP -MF $depbase.Tpo -c -o BESXMLInterface.lo BESXMLInterface.cc &&\
mv -f $depbase.Tpo $depbase.Plo
depbase=`echo BESXMLCommand.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLCommand.lo -MD -MP -MF $depbase.Tpo -c -o BESXMLCommand.lo BESXMLCommand.cc &&\
mv -f $depbase.Tpo $depbase.Plo
depbase=`echo BESXMLUtils.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLUtils.lo -MD -MP -MF $depbase.Tpo -c -o BESXMLUtils.lo BESXMLUtils.cc &&\
mv -f $depbase.Tpo $depbase.Plo
depbase=`echo BESXMLDefaultCommands.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLDefaultCommands.lo -MD -MP -MF $depbase.Tpo -c -o BESXMLDefaultCommands.lo BESXMLDefaultCommands.cc &&\
mv -f $depbase.Tpo $depbase.Plo
depbase=`echo BESXMLShowCommand.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLShowCommand.lo -MD -MP -MF $depbase.Tpo -c -o BESXMLShowCommand.lo BESXMLShowCommand.cc &&\
mv -f $depbase.Tpo $depbase.Plo
depbase=`echo BESXMLShowErrorCommand.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLShowErrorCommand.lo -MD -MP -MF $depbase.Tpo -c -o BESXMLShowErrorCommand.lo BESXMLShowErrorCommand.cc &&\
mv -f $depbase.Tpo $depbase.Plo
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLInterface.lo -MD -MP -MF .deps/BESXMLInterface.Tpo -c BESXMLInterface.cc -fPIC -DPIC -o .libs/BESXMLInterface.o
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLDefaultCommands.lo -MD -MP -MF .deps/BESXMLDefaultCommands.Tpo -c BESXMLDefaultCommands.cc -fPIC -DPIC -o .libs/BESXMLDefaultCommands.o
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLCommand.lo -MD -MP -MF .deps/BESXMLCommand.Tpo -c BESXMLCommand.cc -fPIC -DPIC -o .libs/BESXMLCommand.o
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLUtils.lo -MD -MP -MF .deps/BESXMLUtils.Tpo -c BESXMLUtils.cc -fPIC -DPIC -o .libs/BESXMLUtils.o
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLShowErrorCommand.lo -MD -MP -MF .deps/BESXMLShowErrorCommand.Tpo -c BESXMLShowErrorCommand.cc -fPIC -DPIC -o .libs/BESXMLShowErrorCommand.o
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLShowCommand.lo -MD -MP -MF .deps/BESXMLShowCommand.Tpo -c BESXMLShowCommand.cc -fPIC -DPIC -o .libs/BESXMLShowCommand.o
In file included from BESXMLCommand.cc:37:
BESXMLCommand.h:88:32: error: 'xmlNode' has not been declared
88 | virtual void parse_request(xmlNode *node) = 0;
| ^~~~~~~
make[3]: *** [Makefile:758: BESXMLCommand.lo] Error 1
make[3]: *** Waiting for unfinished jobs....
In file included from BESXMLShowCommand.h:36,
from BESXMLDefaultCommands.cc:42:
BESXMLCommand.h:88:32: error: 'xmlNode' has not been declared
88 | virtual void parse_request(xmlNode *node) = 0;
| ^~~~~~~
BESXMLShowCommand.h:46:32: error: 'xmlNode' has not been declared
46 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLDefaultCommands.cc:43:
BESXMLShowErrorCommand.h:46:32: error: 'xmlNode' has not been declared
46 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLDefaultCommands.cc:44:
BESXMLSetContextCommand.h:46:32: error: 'xmlNode' has not been declared
46 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLDefaultCommands.cc:45:
BESXMLSetContainerCommand.h:61:32: error: 'xmlNode' has not been declared
61 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLDefaultCommands.cc:46:
BESXMLDefineCommand.h:57:62: error: 'xmlNode' has not been declared
57 | void handle_container_element(const std::string &action, xmlNode *node, const std::string &vallues,
| ^~~~~~~
BESXMLDefineCommand.h:72:32: error: 'xmlNode' has not been declared
72 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLDefaultCommands.cc:47:
BESXMLGetCommand.h:56:32: error: 'xmlNode' has not been declared
56 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLDefaultCommands.cc:48:
BESXMLDeleteContainerCommand.h:46:32: error: 'xmlNode' has not been declared
46 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLDefaultCommands.cc:49:
BESXMLDeleteContainersCommand.h:46:32: error: 'xmlNode' has not been declared
46 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLDefaultCommands.cc:50:
BESXMLDeleteDefinitionCommand.h:46:32: error: 'xmlNode' has not been declared
46 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLDefaultCommands.cc:51:
BESXMLDeleteDefinitionsCommand.h:46:32: error: 'xmlNode' has not been declared
46 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLDefaultCommands.cc:53:
ShowPathInfoCommand.h:41:32: error: 'xmlNode' has not been declared
41 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLDefaultCommands.cc:54:
ShowBesKeyCommand.h:47:32: error: 'xmlNode' has not been declared
47 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLDefaultCommands.cc:58:
XMLSetContextsCommand.h:38:32: error: 'xmlNode' has not been declared
38 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
make[3]: *** [Makefile:758: BESXMLDefaultCommands.lo] Error 1
In file included from BESXMLShowErrorCommand.h:36,
from BESXMLShowErrorCommand.cc:35:
BESXMLCommand.h:88:32: error: 'xmlNode' has not been declared
88 | virtual void parse_request(xmlNode *node) = 0;
| ^~~~~~~
BESXMLShowErrorCommand.h:46:32: error: 'xmlNode' has not been declared
46 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLShowErrorCommand.cc:36:
BESXMLUtils.h:49:26: error: 'xmlNode' has not been declared
49 | static void GetProps(xmlNode *node, std::map<std::string, std::string> &props);
| ^~~~~~~
BESXMLUtils.h:50:29: error: 'xmlNode' has not been declared
50 | static void GetNodeInfo(xmlNode *node, std::string &name, std::string &value, std::map<std::string, std::string> &props);
| ^~~~~~~
BESXMLUtils.h:51:12: error: 'xmlNode' does not name a type
51 | static xmlNode * GetFirstChild(xmlNode *node, std::string &child_name, std::string &child_value,
| ^~~~~~~
BESXMLUtils.h:53:12: error: 'xmlNode' does not name a type
53 | static xmlNode * GetNextChild(xmlNode *child_node, std::string &next_name, std::string &next_value,
| ^~~~~~~
BESXMLUtils.h:55:12: error: 'xmlNode' does not name a type
55 | static xmlNode * GetChild(xmlNode *node, const std::string &child_name, std::string &child_value,
| ^~~~~~~
BESXMLShowErrorCommand.cc:66:6: error: variable or field 'parse_request' declared void
66 | void BESXMLShowErrorCommand::parse_request(xmlNode *node)
| ^~~~~~~~~~~~~~~~~~~~~~
BESXMLShowErrorCommand.cc:66:44: error: 'xmlNode' was not declared in this scope
66 | void BESXMLShowErrorCommand::parse_request(xmlNode *node)
| ^~~~~~~
BESXMLShowErrorCommand.cc:66:53: error: 'node' was not declared in this scope
66 | void BESXMLShowErrorCommand::parse_request(xmlNode *node)
| ^~~~
In file included from BESXMLUtils.cc:33:
BESXMLUtils.h:49:26: error: 'xmlNode' has not been declared
49 | static void GetProps(xmlNode *node, std::map<std::string, std::string> &props);
| ^~~~~~~
BESXMLUtils.h:50:29: error: 'xmlNode' has not been declared
50 | static void GetNodeInfo(xmlNode *node, std::string &name, std::string &value, std::map<std::string, std::string> &props);
| ^~~~~~~
BESXMLUtils.h:51:12: error: 'xmlNode' does not name a type
51 | static xmlNode * GetFirstChild(xmlNode *node, std::string &child_name, std::string &child_value,
| ^~~~~~~
BESXMLUtils.h:53:12: error: 'xmlNode' does not name a type
53 | static xmlNode * GetNextChild(xmlNode *child_node, std::string &next_name, std::string &next_value,
| ^~~~~~~
BESXMLUtils.h:55:12: error: 'xmlNode' does not name a type
55 | static xmlNode * GetChild(xmlNode *node, const std::string &child_name, std::string &child_value,
| ^~~~~~~
BESXMLUtils.cc: In static member function 'static void BESXMLUtils::XMLErrorFunc(void*, const char*, ...)':
BESXMLUtils.cc:57:5: error: 'va_start' was not declared in this scope
57 | va_start( args, msg );
| ^~~~~~~~
BESXMLUtils.cc:60:5: error: 'va_end' was not declared in this scope
60 | va_end(args); // Added jhrg 9/17/15
| ^~~~~~
BESXMLUtils.cc: At global scope:
BESXMLUtils.cc:74:6: error: variable or field 'GetProps' declared void
74 | void BESXMLUtils::GetProps(xmlNode *node, map<string, string> &props)
| ^~~~~~~~~~~
BESXMLUtils.cc:74:28: error: 'xmlNode' was not declared in this scope
74 | void BESXMLUtils::GetProps(xmlNode *node, map<string, string> &props)
| ^~~~~~~
BESXMLUtils.cc:74:37: error: 'node' was not declared in this scope
74 | void BESXMLUtils::GetProps(xmlNode *node, map<string, string> &props)
| ^~~~
BESXMLUtils.cc:74:63: error: expected primary-expression before '&' token
74 | void BESXMLUtils::GetProps(xmlNode *node, map<string, string> &props)
| ^
BESXMLUtils.cc:74:64: error: 'props' was not declared in this scope
74 | void BESXMLUtils::GetProps(xmlNode *node, map<string, string> &props)
| ^~~~~
BESXMLUtils.cc:109:6: error: variable or field 'GetNodeInfo' declared void
109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
| ^~~~~~~~~~~
BESXMLUtils.cc:109:31: error: 'xmlNode' was not declared in this scope
109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
| ^~~~~~~
BESXMLUtils.cc:109:40: error: 'node' was not declared in this scope
109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
| ^~~~
BESXMLUtils.cc:109:53: error: expected primary-expression before '&' token
109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
| ^
BESXMLUtils.cc:109:54: error: 'name' was not declared in this scope; did you mean 'tzname'?
109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
| ^~~~
| tzname
BESXMLUtils.cc:109:67: error: expected primary-expression before '&' token
109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
| ^
BESXMLUtils.cc:109:68: error: 'value' was not declared in this scope
109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
| ^~~~~
BESXMLUtils.cc:109:95: error: expected primary-expression before '&' token
109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
| ^
BESXMLUtils.cc:109:96: error: 'props' was not declared in this scope
109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
| ^~~~~
BESXMLUtils.cc:142:1: error: 'xmlNode' does not name a type
142 | xmlNode *
| ^~~~~~~
BESXMLUtils.cc:169:1: error: 'xmlNode' does not name a type
169 | xmlNode *
| ^~~~~~~
BESXMLUtils.cc:195:1: error: 'xmlNode' does not name a type
195 | xmlNode *
| ^~~~~~~
make[3]: *** [Makefile:758: BESXMLShowErrorCommand.lo] Error 1
make[3]: *** [Makefile:758: BESXMLUtils.lo] Error 1
In file included from BESXMLShowCommand.h:36,
from BESXMLShowCommand.cc:35:
BESXMLCommand.h:88:32: error: 'xmlNode' has not been declared
88 | virtual void parse_request(xmlNode *node) = 0;
| ^~~~~~~
BESXMLShowCommand.h:46:32: error: 'xmlNode' has not been declared
46 | virtual void parse_request(xmlNode *node);
| ^~~~~~~
In file included from BESXMLShowCommand.cc:36:
BESXMLUtils.h:49:26: error: 'xmlNode' has not been declared
49 | static void GetProps(xmlNode *node, std::map<std::string, std::string> &props);
| ^~~~~~~
BESXMLUtils.h:50:29: error: 'xmlNode' has not been declared
50 | static void GetNodeInfo(xmlNode *node, std::string &name, std::string &value, std::map<std::string, std::string> &props);
| ^~~~~~~
BESXMLUtils.h:51:12: error: 'xmlNode' does not name a type
51 | static xmlNode * GetFirstChild(xmlNode *node, std::string &child_name, std::string &child_value,
| ^~~~~~~
BESXMLUtils.h:53:12: error: 'xmlNode' does not name a type
53 | static xmlNode * GetNextChild(xmlNode *child_node, std::string &next_name, std::string &next_value,
| ^~~~~~~
BESXMLUtils.h:55:12: error: 'xmlNode' does not name a type
55 | static xmlNode * GetChild(xmlNode *node, const std::string &child_name, std::string &child_value,
| ^~~~~~~
BESXMLShowCommand.cc:61:6: error: variable or field 'parse_request' declared void
61 | void BESXMLShowCommand::parse_request(xmlNode *node)
| ^~~~~~~~~~~~~~~~~
BESXMLShowCommand.cc:61:39: error: 'xmlNode' was not declared in this scope
61 | void BESXMLShowCommand::parse_request(xmlNode *node)
| ^~~~~~~
BESXMLShowCommand.cc:61:48: error: 'node' was not declared in this scope
61 | void BESXMLShowCommand::parse_request(xmlNode *node)
| ^~~~
make[3]: *** [Makefile:758: BESXMLShowCommand.lo] Error 1
In file included from BESXMLInterface.cc:39:
BESXMLCommand.h:88:32: error: 'xmlNode' has not been declared
88 | virtual void parse_request(xmlNode *node) = 0;
| ^~~~~~~
In file included from BESXMLInterface.cc:40:
BESXMLUtils.h:49:26: error: 'xmlNode' has not been declared
49 | static void GetProps(xmlNode *node, std::map<std::string, std::string> &props);
| ^~~~~~~
BESXMLUtils.h:50:29: error: 'xmlNode' has not been declared
50 | static void GetNodeInfo(xmlNode *node, std::string &name, std::string &value, std::map<std::string, std::string> &props);
| ^~~~~~~
BESXMLUtils.h:51:12: error: 'xmlNode' does not name a type
51 | static xmlNode * GetFirstChild(xmlNode *node, std::string &child_name, std::string &child_value,
| ^~~~~~~
BESXMLUtils.h:53:12: error: 'xmlNode' does not name a type
53 | static xmlNode * GetNextChild(xmlNode *child_node, std::string &next_name, std::string &next_value,
| ^~~~~~~
BESXMLUtils.h:55:12: error: 'xmlNode' does not name a type
55 | static xmlNode * GetChild(xmlNode *node, const std::string &child_name, std::string &child_value,
| ^~~~~~~
BESXMLInterface.cc: In member function 'virtual void BESXMLInterface::build_data_request_plan()':
BESXMLInterface.cc:88:5: error: 'xmlDoc' was not declared in this scope
88 | xmlDoc *doc = nullptr;
| ^~~~~~
BESXMLInterface.cc:88:13: error: 'doc' was not declared in this scope
88 | xmlDoc *doc = nullptr;
| ^~~
BESXMLInterface.cc:89:5: error: 'xmlNode' was not declared in this scope
89 | xmlNode *root_element = nullptr;
| ^~~~~~~
BESXMLInterface.cc:89:14: error: 'root_element' was not declared in this scope
89 | xmlNode *root_element = nullptr;
| ^~~~~~~~~~~~
BESXMLInterface.cc:90:14: error: 'current_node' was not declared in this scope
90 | xmlNode *current_node = nullptr;
| ^~~~~~~~~~~~
BESXMLInterface.cc:95:9: error: 'xmlSetGenericErrorFunc' was not declared in this scope
95 | xmlSetGenericErrorFunc((void *) &parseerrors, BESXMLUtils::XMLErrorFunc);
| ^~~~~~~~~~~~~~~~~~~~~~
BESXMLInterface.cc:99:53: error: 'XML_PARSE_NONET' was not declared in this scope
99 | nullptr /* encoding */, XML_PARSE_NONET /* xmlParserOption */);
| ^~~~~~~~~~~~~~~
BESXMLInterface.cc:98:15: error: 'xmlReadMemory' was not declared in this scope
98 | doc = xmlReadMemory(d_xml_document.c_str(), (int)d_xml_document.size(), "" /* base URL */,
| ^~~~~~~~~~~~~
BESXMLInterface.cc:117:24: error: 'xmlDocGetRootElement' was not declared in this scope
117 | root_element = xmlDocGetRootElement(doc);
| ^~~~~~~~~~~~~~~~~~~~
BESXMLInterface.cc:147:39: error: 'XML_ELEMENT_NODE' was not declared in this scope
147 | if (current_node->type == XML_ELEMENT_NODE) {
| ^~~~~~~~~~~~~~~~
BESXMLInterface.cc:228:9: error: 'xmlFreeDoc' was not declared in this scope
228 | xmlFreeDoc(doc);
| ^~~~~~~~~~
BESXMLInterface.cc:229:9: error: 'xmlCleanupParser' was not declared in this scope
229 | xmlCleanupParser();
| ^~~~~~~~~~~~~~~~
BESXMLInterface.cc:233:5: error: 'xmlFreeDoc' was not declared in this scope
233 | xmlFreeDoc(doc);
| ^~~~~~~~~~
BESXMLInterface.cc:245:5: error: 'xmlCleanupParser' was not declared in this scope
245 | xmlCleanupParser();
| ^~~~~~~~~~~~~~~~
make[3]: *** [Makefile:758: BESXMLInterface.lo] Error 1
make[3]: Leaving directory '/home/ruby/rpmbuild/SOURCES/bes-3.21.0/xmlcommand'
make[2]: *** [Makefile:799: all-recursive] Error 1
make[2]: Leaving directory '/home/ruby/rpmbuild/SOURCES/bes-3.21.0/xmlcommand'
make[1]: *** [Makefile:773: all-recursive] Error 1
make[1]: Leaving directory '/home/ruby/rpmbuild/SOURCES/bes-3.21.0'
make: *** [Makefile:548: all] Error 2
detailed build log: https://gist.github.com/topazus/a369486a6b061e9c3a8aac31c015bf71
I get the following messages on CentOS:
tar: Ignoring unknown extended header keyword `SCHILY.dev`
tar: Ignoring unknown extended header keyword `SCHILY.ino`
tar: Ignoring unknown extended header keyword `SCHILY.nlink`
I think the source package is made from Mac OS X and causes the above output.
In the get_dmrpp
script's dap4_array_test
function, it selects a random number in order to test a variable of a given type. It calls dap4_array_dim_counter
which selects the variable, but that function assumes that the match_num
variable starts at 1. So if 0 is passed, it won't select a random variable and will result in an error.
I'm looking to update the Fedora bes package to 3.20.10 but am running into the following issue:
+ /usr/lib/rpm/check-rpaths
*******************************************************************************
*
* WARNING: 'check-rpaths' detected a broken RPATH OR RUNPATH and will cause
* 'rpmbuild' to fail. To ignore these errors, you can set the
* '$QA_RPATHS' environment variable which is a bitmask allowing the
* values below. The current value of QA_RPATHS is 0x0000.
*
* 0x0001 ... standard RPATHs (e.g. /usr/lib); such RPATHs are a minor
* issue but are introducing redundant searchpaths without
* providing a benefit. They can also cause errors in multilib
* environments.
* 0x0002 ... invalid RPATHs; these are RPATHs which are neither absolute
* nor relative filenames and can therefore be a SECURITY risk
* 0x0004 ... insecure RPATHs; these are relative RPATHs which are a
* SECURITY risk
* 0x0008 ... the special '$ORIGIN' RPATHs are appearing after other
* RPATHs; this is just a minor issue but usually unwanted
* 0x0010 ... the RPATH is empty; there is no reason for such RPATHs
* and they cause unneeded work while loading libraries
* 0x0020 ... an RPATH references '..' of an absolute path; this will break
* the functionality when the path before '..' is a symlink
*
*
* Examples:
* - to ignore standard and empty RPATHs, execute 'rpmbuild' like
* $ QA_RPATHS=$(( 0x0001|0x0010 )) rpmbuild my-package.src.rpm
* - to check existing files, set $RPM_BUILD_ROOT and execute check-rpaths like
* $ RPM_BUILD_ROOT=<top-dir> /usr/lib/rpm/check-rpaths
*
*******************************************************************************
ERROR 0002: file '/usr/bin/build_dmrpp' contains an invalid '/builddir/build/BUILD/bes-3.20.10/dap/.libs' in [/builddir/build/BUILD/bes-3.20.10/dap/.libs:/builddir/build/BUILD/bes-3.20.10/dispatch/.libs:/usr/lib64/bes]
ERROR 0002: file '/usr/bin/build_dmrpp' contains an invalid '/builddir/build/BUILD/bes-3.20.10/dispatch/.libs' in [/builddir/build/BUILD/bes-3.20.10/dap/.libs:/builddir/build/BUILD/bes-3.20.10/dispatch/.libs:/usr/lib64/bes]
This appears to arise due to build_drmpp being linked to a module:
make[4]: Entering directory '/builddir/build/BUILD/bes-3.20.10/modules/dmrpp_module'
/bin/sh ../../libtool --tag=CXX --mode=link g++ -Wno-vla-extension -Wno-inconsistent-missing-override -g -O2 --pedantic --std=c++11 -L../../dap -static -ldap_module -dynamic -L/usr/lib64/libdap -o build_dmrpp build_dmrpp-DMRpp.o build_dmrpp-DmrppCommon.o build_dmrpp-Chunk.o build_dmrpp-CurlHandlePool.o build_dmrpp-DmrppByte.o build_dmrpp-DmrppArray.o build_dmrpp-DmrppFloat32.o build_dmrpp-DmrppFloat64.o build_dmrpp-DmrppInt16.o build_dmrpp-DmrppInt32.o build_dmrpp-DmrppInt64.o build_dmrpp-DmrppInt8.o build_dmrpp-DmrppUInt16.o build_dmrpp-DmrppUInt32.o build_dmrpp-DmrppUInt64.o build_dmrpp-DmrppStr.o build_dmrpp-DmrppStructure.o build_dmrpp-DmrppUrl.o build_dmrpp-DmrppD4Enum.o build_dmrpp-DmrppD4Group.o build_dmrpp-DmrppD4Opaque.o build_dmrpp-DmrppD4Sequence.o build_dmrpp-DmrppTypeFactory.o build_dmrpp-DmrppParserSax2.o build_dmrpp-DmrppMetadataStore.o build_dmrpp-CredentialsManager.o build_dmrpp-AccessCredentials.o build_dmrpp-NgapS3Credentials.o build_dmrpp-SuperChunk.o build_dmrpp-awsv4.o build_dmrpp-DMZ.o build_dmrpp-DmrppRequestHandler.o build_dmrpp-build_dmrpp.o build_dmrpp-h5common.o -L../../dispatch -lbes_dispatch -L../../http -lbes_http -lcurl -luuid -ltirpc -L. -lhdf5 -ldl -ldapserver -ldap -ldapclient -ldap -L/usr/local/opt/openssl/lib -lcrypto -lxml2 -lz -lcurl -luuid -ltirpc
*** Warning: Linking the executable build_dmrpp against the loadable module
*** libdap_module.so is not portable!
libtool: link: g++ -Wno-vla-extension -Wno-inconsistent-missing-override -g -O2 --pedantic --std=c++11 -dynamic -o build_dmrpp build_dmrpp-DMRpp.o build_dmrpp-DmrppCommon.o build_dmrpp-Chunk.o build_dmrpp-CurlHandlePool.o build_dmrpp-DmrppByte.o build_dmrpp-DmrppArray.o build_dmrpp-DmrppFloat32.o build_dmrpp-DmrppFloat64.o build_dmrpp-DmrppInt16.o build_dmrpp-DmrppInt32.o build_dmrpp-DmrppInt64.o build_dmrpp-DmrppInt8.o build_dmrpp-DmrppUInt16.o build_dmrpp-DmrppUInt32.o build_dmrpp-DmrppUInt64.o build_dmrpp-DmrppStr.o build_dmrpp-DmrppStructure.o build_dmrpp-DmrppUrl.o build_dmrpp-DmrppD4Enum.o build_dmrpp-DmrppD4Group.o build_dmrpp-DmrppD4Opaque.o build_dmrpp-DmrppD4Sequence.o build_dmrpp-DmrppTypeFactory.o build_dmrpp-DmrppParserSax2.o build_dmrpp-DmrppMetadataStore.o build_dmrpp-CredentialsManager.o build_dmrpp-AccessCredentials.o build_dmrpp-NgapS3Credentials.o build_dmrpp-SuperChunk.o build_dmrpp-awsv4.o build_dmrpp-DMZ.o build_dmrpp-DmrppRequestHandler.o build_dmrpp-build_dmrpp.o build_dmrpp-h5common.o -L../../dap /builddir/build/BUILD/bes-3.20.10/dap/.libs/libdap_module.so -L/usr/lib64/libdap -L../../dispatch /builddir/build/BUILD/bes-3.20.10/dispatch/.libs/libbes_dispatch.so -L../../http /builddir/build/BUILD/bes-3.20.10/http/.libs/libbes_http.a -L../dispatch ../../dispatch/.libs/libbes_dispatch.so -lbz2 -lpthread -L/usr/local/opt/openssl/lib -L. -lhdf5 -ldl -ldapserver -ldapclient -ldap -lcrypto -lxml2 -lz -lcurl -luuid -ltirpc -Wl,-rpath -Wl,/builddir/build/BUILD/bes-3.20.10/dap/.libs -Wl,-rpath -Wl,/builddir/build/BUILD/bes-3.20.10/dispatch/.libs -Wl,-rpath -Wl,/usr/lib64/bes
make[4]: Leaving directory '/builddir/build/BUILD/bes-3.20.10/modules/dmrpp_module'
which libtool notes is "not portable".
diff -ur bes-3.21.0-46.old/http/CurlUtils.cc bes-3.21.0-46.new/http/CurlUtils.cc
--- bes-3.21.0-46.old/http/CurlUtils.cc 2024-01-23 23:28:51.000000000 +0000
+++ bes-3.21.0-46.new/http/CurlUtils.cc 2024-02-23 13:19:11.556704797 +0000
@@ -27,6 +27,7 @@
#include <unistd.h>
#include <fcntl.h>
#include <ctime>
+#include <cstring>
#include <curl/curl.h>
diff -ur bes-3.21.0-46.old/http/RemoteResource.cc bes-3.21.0-46.new/http/RemoteResource.cc
--- bes-3.21.0-46.old/http/RemoteResource.cc 2024-01-23 23:28:51.000000000 +0000
+++ bes-3.21.0-46.new/http/RemoteResource.cc 2024-02-23 13:19:39.630855718 +0000
@@ -28,6 +28,7 @@
#include <cstdio>
#include <unistd.h>
+#include <cstring>
#include <sstream>
#include <string>
diff -ur bes-3.21.0-46.old/http/tests/remote_resource_tester.cc bes-3.21.0-46.new/http/tests/remote_resource_tester.cc
--- bes-3.21.0-46.old/http/tests/remote_resource_tester.cc 2024-01-23 23:28:51.000000000 +0000
+++ bes-3.21.0-46.new/http/tests/remote_resource_tester.cc 2024-02-23 13:20:12.830034191 +0000
@@ -6,6 +6,7 @@
#include <iostream>
#include <string>
+#include <cstring>
#include <sys/stat.h>
#include <unistd.h>
Hi,
I am trying to build the bes module from source using Build Steps guide using hyrax-dependencies. Previously it worked fine but with latest git code pull it gives error while running make command.
**DmrppArray.cc: In member function ‘void dmrpp::DmrppArray::multi_finish(CURLM*, std::vector<dmrpp::H4ByteStream, std::allocator<dmrpp::H4ByteStream> >*)’:
DmrppArray.cc:475: error: ‘curl_multi_wait’ was not declared in this scope
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppModule.lo -MD -MP -MF .deps/DmrppModule.Tpo -c DmrppModule.cc -o DmrppModule.o >/dev/null 2>&1
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT H4ByteStream.lo -MD -MP -MF .deps/H4ByteStream.Tpo -c H4ByteStream.cc -o H4ByteStream.o >/dev/null 2>&1
make[4]: *** [DmrppArray.lo] Error 1
make[4]: *** Waiting for unfinished jobs....
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppCommon.lo -MD -MP -MF .deps/DmrppCommon.Tpo -c DmrppCommon.cc -o DmrppCommon.o >/dev/null 2>&1
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppByte.lo -MD -MP -MF .deps/DmrppByte.Tpo -c DmrppByte.cc -o DmrppByte.o >/dev/null 2>&1
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppFloat64.lo -MD -MP -MF .deps/DmrppFloat64.Tpo -c DmrppFloat64.cc -o DmrppFloat64.o >/dev/null 2>&1
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppInt16.lo -MD -MP -MF .deps/DmrppInt16.Tpo -c DmrppInt16.cc -o DmrppInt16.o >/dev/null 2>&1
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppFloat32.lo -MD -MP -MF .deps/DmrppFloat32.Tpo -c DmrppFloat32.cc -o DmrppFloat32.o >/dev/null 2>&1
mv -f .deps/DmrppModule.Tpo .deps/DmrppModule.Plo
mv -f .deps/DmrppCommon.Tpo .deps/DmrppCommon.Plo
mv -f .deps/H4ByteStream.Tpo .deps/H4ByteStream.Plo
mv -f .deps/DmrppInt16.Tpo .deps/DmrppInt16.Plo
mv -f .deps/DmrppByte.Tpo .deps/DmrppByte.Plo
mv -f .deps/DmrppFloat64.Tpo .deps/DmrppFloat64.Plo
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppRequestHandler.lo -MD -MP -MF .deps/DmrppRequestHandler.Tpo -c DmrppRequestHandler.cc -o DmrppRequestHandler.o >/dev/null 2>&1
mv -f .deps/DmrppFloat32.Tpo .deps/DmrppFloat32.Plo
mv -f .deps/DmrppRequestHandler.Tpo .deps/DmrppRequestHandler.Plo
make[4]: Leaving directory `/hyrax-git/bes/modules/dmrpp_module'
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory `/hyrax-git/bes/modules/dmrpp_module'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/hyrax-git/bes/modules'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/hyrax-git/bes'
make: *** [all] Error 2**
Building without HDFEOS2 works fine.
Building with it gives this -fPIC error message. However, I'm sure that the HDFEOS library I'm pointing to was built with -fPIC. Also @jgallagher59701 reported this to me as well, but I thought it was because he was pointing to a non-relocatable library. But perhaps not...
/bin/bash ../../libtool --tag=CXX --mode=link g++ -I../../hdfclass -g -O2 --pedantic --std=c++11 -avoid-version -module -o libhdf4_module.la -rpath /usr/local/bes-3.20.7/lib/bes hdfeos.tab.lo lex.hdfeos.lo HDFArray.lo HDFGrid.lo HDFSequence.lo HDFUInt32.lo HDFByte.lo HDFInt16.lo HDFStr.lo HDFUrl.lo HDFFloat32.lo HDFInt32.lo HDFStructure.lo HDFFloat64.lo HDFUInt16.lo HDFTypeFactory.lo HDFCFUtil.lo BESH4MCache.lo HDFCFStr.lo HDFEOS2.lo HDFEOS2CFStr.lo HDFSP.lo HDFSPArrayGeoField.lo HDFSPArrayAddCVField.lo HDFSPArrayMissField.lo HDFSPArray_RealField.lo HDFSPArray_VDField.lo HDFCFStrField.lo HDFEOS2CFStrField.lo HDFEOS2Array_RealField.lo HDFEOS2ArrayGridGeoField.lo HDFEOS2ArraySwathGeoField.lo HDFEOS2ArrayMissField.lo HDFEOS2ArraySwathDimMapField.lo HDFEOS2ArraySwathGeoDimMapExtraField.lo HDFEOS2ArraySwathGeoMultiDimMapField.lo HDFEOS2GeoCFProj.lo HDFEOS2GeoCF1D.lo HE2CF.lo misr_init.lo misrinv.lo dhdferr.lo hdfdesc.lo hc2dap.lo hdfutil.lo HDF4RequestHandler.lo HDF4Module.lo hdfclass/libhdfclass.la -L/usr/local/hdfeos2.19_fPIC/lib -lhdfeos -lGctp -lmfhdf -ldf -ljpeg -lz -lsz -ldapserver -ldap -ldapclient -ldap -lcurl -luuid
libtool: link: g++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginS.o .libs/hdfeos.tab.o .libs/lex.hdfeos.o .libs/HDFArray.o .libs/HDFGrid.o .libs/HDFSequence.o .libs/HDFUInt32.o .libs/HDFByte.o .libs/HDFInt16.o .libs/HDFStr.o .libs/HDFUrl.o .libs/HDFFloat32.o .libs/HDFInt32.o .libs/HDFStructure.o .libs/HDFFloat64.o .libs/HDFUInt16.o .libs/HDFTypeFactory.o .libs/HDFCFUtil.o .libs/BESH4MCache.o .libs/HDFCFStr.o .libs/HDFEOS2.o .libs/HDFEOS2CFStr.o .libs/HDFSP.o .libs/HDFSPArrayGeoField.o .libs/HDFSPArrayAddCVField.o .libs/HDFSPArrayMissField.o .libs/HDFSPArray_RealField.o .libs/HDFSPArray_VDField.o .libs/HDFCFStrField.o .libs/HDFEOS2CFStrField.o .libs/HDFEOS2Array_RealField.o .libs/HDFEOS2ArrayGridGeoField.o .libs/HDFEOS2ArraySwathGeoField.o .libs/HDFEOS2ArrayMissField.o .libs/HDFEOS2ArraySwathDimMapField.o .libs/HDFEOS2ArraySwathGeoDimMapExtraField.o .libs/HDFEOS2ArraySwathGeoMultiDimMapField.o .libs/HDFEOS2GeoCFProj.o .libs/HDFEOS2GeoCF1D.o .libs/HE2CF.o .libs/misr_init.o .libs/misrinv.o .libs/dhdferr.o .libs/hdfdesc.o .libs/hc2dap.o .libs/hdfutil.o .libs/HDF4RequestHandler.o .libs/HDF4Module.o -Wl,--whole-archive hdfclass/.libs/libhdfclass.a -Wl,--no-whole-archive -L/usr/local/hdf-4.2.15_fPIC/lib -L/usr/local/hdfeos2.19_fPIC/lib /usr/local/hdfeos2.19_fPIC/lib/libhdfeos.a /usr/local/hdfeos2.19_fPIC/lib/libGctp.a /usr/local/hdf-4.2.15_fPIC/lib/libmfhdf.a /usr/local/hdf-4.2.15_fPIC/lib/libdf.a /home/ed/hyrax/build/deps/lib/libmfhdf.a -L/home/ed/hyrax/build/deps/lib /home/ed/hyrax/build/deps/lib/libdf.a -ljpeg -lz -lsz -ldapserver -ldapclient -ldap /usr/lib/x86_64-linux-gnu/libcurl.so -luuid -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/9/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/9/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crtn.o -g -O2 -pthread -Wl,-soname -Wl,libhdf4_module.so -o .libs/libhdf4_module.so
/usr/bin/ld: /usr/local/hdfeos2.19_fPIC/lib/libhdfeos.a(EHapi.o): relocation R_X86_64_PC32 against symbol `EHXtypeTable' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:750: libhdf4_module.la] Error 1
make[4]: Leaving directory '/home/ed/bes/modules/hdf4_handler'
make[3]: *** [Makefile:888: all-recursive] Error 1
make[3]: Leaving directory '/home/ed/bes/modules/hdf4_handler'
make[2]: *** [Makefile:504: all-recursive] Error 1
make[2]: Leaving directory '/home/ed/bes/modules'
make[1]: *** [Makefile:759: all-recursive] Error 1
make[1]: Leaving directory '/home/ed/bes'
make: *** [Makefile:542: all] Error 2
Getting some warnings about the subdir-objects setting.
This setting causes object files to be always placed in the same build dir as their source, instead of the build dir of the Makefile.am that is using them. (This is the same directory unless the Makefile.am includes source files from other directories, as happens in several places in bes.) This is documented here: https://www.gnu.org/software/automake/manual/html_node/Program-and-Library-Variables.html
IIRC this is a result of changes in automake defaults. In olden times, subdir-objects was on by default, but now it is off by default.
This is fixed by adding subdir-objects to the list of options for AM_INIT_AUTOMAKE in configure.ac.
functions/unit-tests/Makefile.am:140: warning: source file '../RangeFunction.cc' is in a subdirectory,
functions/unit-tests/Makefile.am:140: but option 'subdir-objects' is disabled
automake: warning: possible forward-incompatibility.
automake: At least a source file is in a subdirectory, but the 'subdir-objects'
automake: automake option hasn't been enabled. For now, the corresponding output
automake: object file(s) will be placed in the top-level directory. However,
automake: this behaviour will change in future Automake versions: they will
automake: unconditionally cause object files to be placed in the same subdirectory
automake: of the corresponding sources.
automake: You are advised to start using 'subdir-objects' option throughout your
automake: project, to avoid future incompatibilities.
functions/unit-tests/Makefile.am:132: warning: source file '../scale_util.cc' is in a subdirectory,
functions/unit-tests/Makefile.am:132: but option 'subdir-objects' is disabled
functions/unit-tests/Makefile.am:136: warning: source file '../scale_util.cc' is in a subdirectory,
functions/unit-tests/Makefile.am:136: but option 'subdir-objects' is disabled
modules/dmrpp_module/Makefile.am:78: warning: source file '$(srcdir)/../hdf5_handler/h5common.cc' is in a subdirectory,
modules/dmrpp_module/Makefile.am:78: but option 'subdir-objects' is disabled
modules/dmrpp_module/unit-tests/Makefile.am:127: warning: source file '$(top_srcdir)/modules/read_test_baseline.cc' is in a subdirectory,
modules/dmrpp_module/unit-tests/Makefile.am:127: but option 'subdir-objects' is disabled
modules/dmrpp_module/unit-tests/Makefile.am:130: warning: source file '$(top_srcdir)/modules/read_test_baseline.cc' is in a subdirectory,
modules/dmrpp_module/unit-tests/Makefile.am:130: but option 'subdir-objects' is disabled
The HDFEOS2 library is for NASA HDF4 files, like MODIS.
As discussed in this issue (SpatioTemporal/STAREmaster#111), there are two different versions of HDFEOS2 that should be accomodated, 2.19 and 2.20. Unfortunately there is some const changes which interact badly with the hdf4_handler file HDFEOS2.cc.
In HDFEOS2.cc there is code that passes some of the HDFEOS2 functions as function pointers to a function that calls the HDFEOS2 function. For example:
vector<string> gridlist;
if (!Utility::ReadNamelist(file->path.c_str(), GDinqgrid, gridlist)) {
delete file;
throw1("Grid ReadNamelist failed.");
}
This is calling the Utility::ReadNamelist() function, which is also in HDFEOS2.cc:
// Read name list from the EOS2 APIs and store names into a vector
bool Utility::ReadNamelist(const char *path,
int32 (*inq)(const char *, char *, int32 *),
vector<string> &names)
{
char *fname = const_cast<char *>(path);
int32 bufsize;
int numobjs;
if ((numobjs = inq(fname, NULL, &bufsize)) == -1) return false;
if (numobjs > 0) {
vector<char> buffer;
buffer.resize(bufsize + 1);
if (inq(fname, &buffer[0], &bufsize) == -1) return false;
HDFCFUtil::Split(&buffer[0], bufsize, ',', names);
}
return true;
}
Note that the second (unnamed) parameter to ReadNamelis() is a function pointer.
This ReadNamelist function is called only 3 times, for functions GDinqgrid(), SWinqswath(), and PTinqpoint().
The challenge here is that one of these functions, SWinqswath(), has a different first parameter in 2.19 and 2.20. In 2.20 it adds a const.
geogrid(Var, ..., )
works as expected when Var
is a Grid
array. Var
contains a "copy" of the coordinate values lat
and lon
. Grid
s are part of the DAP2 model but were dropped out of the DAP4 model.
When Var
is an Array
geogrid(Var, ...)
fails to find the latitude and longitude arrays it needs.
The following errs (not a very helpful error)
https://oceandata.sci.gsfc.nasa.gov/opendap/MODISA/L3SMI/2002/0101/AQUA_MODIS.20020101_20021231.L3m.YR.RRS.Rrs_555.4km.nc.dap.nc4?dap4.function=geogrid(/Rrs_555,35,-121,32,-116)
@jgallagher59701 reached to this conclusion regarding in OPeNDAP's Discourse Thread
Although the (HDF5) handler in Hyrax 1.13.3 correctly presents the dimensions, a .nc4 download changes them to 'phony_dim_' in download file - this is in fact observed in other data like Merra2 as well. The 3IMERGM data is located here:
https://gpm1.gesdisc.eosdis.nasa.gov/data/s4pa/GPM_L3/GPM_3IMERGM.03/
The other issue with 3IMERG is that an 'nc_' is prepended to the download filename, a very undesirable feature to us. I recognize that both issues could be related to the netcdf library.
DapUtils.cc: In function 'void dap_utils::throw_for_dap4_typed_vars_or_attrs(libdap::DDS*, const std::string&, unsigned int)':
DapUtils.cc:174:13: error: 'class libdap::DDS' has no member named 'is_dap4_projected'
174 | if(dds->is_dap4_projected(inventory)){
| ^~~~~~~~~~~~~~~~~
DapUtils.cc: In function 'void dap_utils::throw_for_dap4_typed_attrs(libdap::DAS*, const std::string&, unsigned int)':
DapUtils.cc:189:41: error: 'class libdap::AttrTable' has no member named 'has_dap4_types'
189 | if(das->get_top_level_attributes()->has_dap4_types("/",inventory)){
| ^~~~~~~~~~~~~~
DapUtils.cc: In function 'uint64_t dap_utils::crsaibv_process_variable(libdap::BaseType*, uint64_t, std::vector<std::__cxx11::basic_string<char> >&)':
DapUtils.cc:316:35: error: 'class libdap::BaseType' has no member named 'width_ll'; did you mean 'width'?
316 | uint64_t vsize = var->width_ll(true);
| ^~~~~~~~
| width
DapUtils.cc: In function 'uint64_t dap_utils::compute_response_size_and_inv_big_vars(const libdap::D4Group*, uint64_t, std::vector<std::__cxx11::basic_string<char> >&)':
DapUtils.cc:384:37: error: 'const class libdap::D4Group' has no member named 'groups'; did you mean 'std::vector<libdap::D4Group*> libdap::D4Group::d_groups'? (not accessible from this context)
384 | for (const auto child_grp: grp->groups()) {
| ^~~~~~
In file included from DapUtils.cc:34:
/usr/include/libdap/D4Group.h:60:22: note: declared private here
60 | vector<D4Group*> d_groups;
| ^~~~~~~~
make[3]: *** [Makefile:815: DapUtils.lo] Error 1
Hi Folks,
Our Capstone team (@rrimer07, @hemphilc and @RiverHendriksen) have encountered a duplicate definitions warning whilst attempting to configure and install bes from source.
Having discussed this, we think it relates to the duplicate definition of SUBDIRS in bes/modules/Makefile.am.
Can someone scope this out and let us know? If you can reproduce the warning, let us know and we will submit a PR. Thanks
In the dmrpp_module/testsuite/bes.conf.in
file both the runtime loadable modules
libdmrpp_module.so
libbuilddmrpp_module.so
are loaded into besstandalone
from the bes.conf
file in the modules/dmrpp_module /testsuite
directory. This causes ODR violations to be detected by ASAN. The first error encountered is the redefinition of http/HttpError.cc
The obvious fix here is to simply eliminate the use of libbuilddmrpp_module.so
from the bes.conf.in
because it's not used in these tests. I removed it and ASAN was happy and all the tests behaved as expected.
But it is an outstanding question as to why this ODR violation happened given that we load lots of modules. Is this something we need to address in how we structure our build/deployment?
Inquiring minds want to know.
It currently tests Float32, but could be expanded to other types.
I downloaded bes_3.14.0-2.NASA_x86_64.deb
from OPeNDAP's experimental debian package index, but ran into a dependency issue when trying to install it with gdebi
. Apparently it requires the package libbz2
, which is not available in the default repos for Ubuntu 14.04 (Trusty Tahr).
There is, however, a package named libbz2-1.0
available for Trusty Tahr; maybe the dependency could be changed to this in the published .deb?
Below is my output showing the failed installation:
~$ sudo gdebi bes_3.14.0-2.NASA_x86_64.deb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Building data structures... Done
Building data structures... Done
This package is uninstallable
Dependency is not satisfiable: libbz2
The development package, bes-devel_3.14.0-2.NASA_x86_64.deb
, installed without issue.
When compiled with --enable-asan the fileout_netcdf handler generates 4 test failures
207: bescmd/vdata_test.hdf.dmrpp.bescmd FAILED (testsuite.at:320)
210: bescmd/vdata_packed_linked_blocks.hdf.dmrpp.bescmd FAILED (testsuite.at:323)
216: bescmd/PointFile.hdf.dmrpp.bescmd FAILED (testsuite.at:329)
217: bescmd/vdata_test.hdf.bescmd FAILED (testsuite.at:333)
218: bescmd/vdata_packed_linked_blocks.hdf.bescmd FAILED (testsuite.at:334)
These errors are all of the same nature: runtime error: load of misaligned address
Is this a design issue arising from using void * or is this a usage issue in the way that the fileout_netcdf is allocating memory?
Or is this problem caused by allocating variable memory as byte
/char
rather than unit32_t
etc.??
Examine and fix if feasible.
Here is the verbose test output for the failed tests:
[-bash: ~/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests] ./testsuite -v 207 210 216 217 218
## -------------------------------------------------------------- ##
## bes 3.21.0 test suite: modules/fileout_netcdf/tests testsuite. ##
## -------------------------------------------------------------- ##
207. testsuite.at:320: testing bescmd/vdata_test.hdf.dmrpp.bescmd ...
COMMAND: besstandalone -c bes.nc4.grp.conf -i bescmd/vdata_test.hdf.dmrpp.bescmd
./testsuite.at:320: besstandalone -c $abs_builddir/$bes_conf -i $input > test.nc
--- /dev/null 2024-05-30 11:28:15
+++ /Users/ndp/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests/testsuite.dir/at-groups/207/stderr 2024-05-30 11:28:16
@@ -0,0 +1,25 @@
+Float32.cc:187:13: runtime error: load of misaligned address 0x611000006a86 for type 'dods_float32' (aka 'float'), which requires 4 byte alignment
+0x611000006a86: note: pointer points here
+ 8e 3f 00 00 7b 14 8e 3f 41 61 30 3f 80 00 00 40 00 00 00 40 0e 14 7b 00 01 40 0e 14 7b 42 62 31
+ ^
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Float32.cc:187:13 in
+DmrppStructure.cc:154:26: runtime error: load of misaligned address 0x611000006a99 for type 'dods_uint16' (aka 'unsigned short'), which requires 2 byte alignment
+0x611000006a99: note: pointer points here
+ 7b 14 0e 40 00 01 40 0e 14 7b 42 62 31 3f 80 00 00 40 00 00 00 40 55 1e b8 00 02 40 55 1e b8 43
+ ^
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior DmrppStructure.cc:154:26 in
+DmrppStructure.cc:154:26: runtime error: load of misaligned address 0x611000006a99 for type 'dods_uint16' (aka 'unsigned short'), which requires 2 byte alignment
+0x611000006a99: note: pointer points here
+ 7b 14 0e 40 00 01 40 0e 14 7b 42 62 31 3f 80 00 00 40 00 00 00 40 55 1e b8 00 02 40 55 1e b8 43
+ ^
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior DmrppStructure.cc:154:26 in
+DmrppStructure.cc:154:17: runtime error: store to misaligned address 0x611000006a99 for type 'dods_uint16' (aka 'unsigned short'), which requires 2 byte alignment
+0x611000006a99: note: pointer points here
+ 7b 14 0e 40 00 01 40 0e 14 7b 42 62 31 3f 80 00 00 40 00 00 00 40 55 1e b8 00 02 40 55 1e b8 43
+ ^
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior DmrppStructure.cc:154:17 in
+Int16.cc:180:13: runtime error: load of misaligned address 0x611000006a99 for type 'dods_int16' (aka 'short'), which requires 2 byte alignment
+0x611000006a99: note: pointer points here
+ 7b 14 0e 40 01 00 40 0e 14 7b 42 62 31 3f 80 00 00 40 00 00 00 40 55 1e b8 00 02 40 55 1e b8 43
+ ^
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Int16.cc:180:13 in
207. testsuite.at:320: FAILED (testsuite.at:320)
210. testsuite.at:323: testing bescmd/vdata_packed_linked_blocks.hdf.dmrpp.bescmd ...
COMMAND: besstandalone -c bes.nc4.grp.conf -i bescmd/vdata_packed_linked_blocks.hdf.dmrpp.bescmd
./testsuite.at:323: besstandalone -c $abs_builddir/$bes_conf -i $input > test.nc
--- /dev/null 2024-05-30 11:28:15
+++ /Users/ndp/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests/testsuite.dir/at-groups/210/stderr 2024-05-30 11:28:17
@@ -0,0 +1,5 @@
+Float32.cc:187:13: runtime error: load of misaligned address 0x60300002a0d6 for type 'dods_float32' (aka 'float'), which requires 4 byte alignment
+0x60300002a0d6: note: pointer points here
+ 8e 3f 00 00 7b 14 0e 40 00 01 3f 8e 14 7b 00 00 40 0e 14 7b 00 01 00 00 00 00 00 00 00 00 02 02
+ ^
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Float32.cc:187:13 in
210. testsuite.at:323: FAILED (testsuite.at:323)
216. testsuite.at:329: testing bescmd/PointFile.hdf.dmrpp.bescmd ...
COMMAND: besstandalone -c bes.nc4.grp.conf -i bescmd/PointFile.hdf.dmrpp.bescmd
./testsuite.at:329: besstandalone -c $abs_builddir/$bes_conf -i $input > test.nc
--- /dev/null 2024-05-30 11:28:15
+++ /Users/ndp/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests/testsuite.dir/at-groups/216/stderr 2024-05-30 11:28:18
@@ -0,0 +1,5 @@
+Float64.cc:185:13: runtime error: load of misaligned address 0x61400000465c for type 'dods_float64' (aka 'double'), which requires 8 byte alignment
+0x61400000465c: note: pointer points here
+ 43 4f 00 00 9a 99 99 25 1c 35 86 41 40 7f 5c 29 40 90 a3 d7 40 16 66 66 40 21 47 ae 43 4f 32 00
+ ^
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Float64.cc:185:13 in
216. testsuite.at:329: FAILED (testsuite.at:329)
217. testsuite.at:333: testing bescmd/vdata_test.hdf.bescmd ...
COMMAND: besstandalone -c bes.nc4.grp.conf -i bescmd/vdata_test.hdf.bescmd
./testsuite.at:333: besstandalone -c $abs_builddir/$bes_conf -i $input > test.nc
--- /dev/null 2024-05-30 11:28:18
+++ /Users/ndp/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests/testsuite.dir/at-groups/217/stderr 2024-05-30 11:28:18
@@ -0,0 +1,10 @@
+Float32.cc:187:13: runtime error: load of misaligned address 0x611000006f86 for type 'dods_float32' (aka 'float'), which requires 4 byte alignment
+0x611000006f86: note: pointer points here
+ 8e 3f 00 00 7b 14 8e 3f 41 61 30 00 00 80 3f 00 00 00 40 7b 14 0e 40 01 00 7b 14 0e 40 42 62 31
+ ^
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Float32.cc:187:13 in
+Int16.cc:180:13: runtime error: load of misaligned address 0x611000006f99 for type 'dods_int16' (aka 'short'), which requires 2 byte alignment
+0x611000006f99: note: pointer points here
+ 7b 14 0e 40 01 00 7b 14 0e 40 42 62 31 00 00 80 3f 00 00 00 40 b8 1e 55 40 02 00 b8 1e 55 40 43
+ ^
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Int16.cc:180:13 in
217. testsuite.at:333: FAILED (testsuite.at:333)
218. testsuite.at:334: testing bescmd/vdata_packed_linked_blocks.hdf.bescmd ...
COMMAND: besstandalone -c bes.nc4.grp.conf -i bescmd/vdata_packed_linked_blocks.hdf.bescmd
./testsuite.at:334: besstandalone -c $abs_builddir/$bes_conf -i $input > test.nc
--- /dev/null 2024-05-30 11:28:19
+++ /Users/ndp/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests/testsuite.dir/at-groups/218/stderr 2024-05-30 11:28:19
@@ -0,0 +1,5 @@
+Float32.cc:187:13: runtime error: load of misaligned address 0x60300002a8e6 for type 'dods_float32' (aka 'float'), which requires 4 byte alignment
+0x60300002a8e6: note: pointer points here
+ 8e 3f 00 00 7b 14 0e 40 01 00 7b 14 8e 3f 00 00 7b 14 0e 40 01 00 00 00 00 00 00 00 00 00 03 02
+ ^
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Float32.cc:187:13 in
218. testsuite.at:334: FAILED (testsuite.at:334)
## ------------- ##
## Test results. ##
## ------------- ##
ERROR: All 5 tests were run,
5 failed unexpectedly.
## -------------------------- ##
## testsuite.log was created. ##
## -------------------------- ##
Please send `modules/fileout_netcdf/tests/testsuite.log' and all information you think might help:
To: <[email protected]>
Subject: [bes 3.21.0] testsuite: 207 210 216 217 218 failed
You may investigate any problem if you feel able to do so, in which
case the test suite provides a good starting point. Its output may
be found below `modules/fileout_netcdf/tests/testsuite.dir'.
[-bash: ~/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests]
This refers to http://test.opendap.org/opendap/AIRS/QC_AIRS-1.2011.02.13.090.L2.RetStd.v5.2.2.0.G11045052855.hdf.covjson.
As far as I see, all the "parameters" listed in the result, e.g. AIRSXTrack
or StdPressureLay
are not parameters but should be domain axes. The actual parameters, looking at http://test.opendap.org/opendap/AIRS/QC_AIRS-1.2011.02.13.090.L2.RetStd.v5.2.2.0.G11045052855.hdf.html, like O3_Resid_Ratio
or TCldTopStd
are all missing.
Just looking at one of the ranges objects of the output makes this mismatch obvious:
"AIRSXTrack": {
"type": "NdArray",
"dataType": "integer",
"axisNames": ["y", "x"],
"shape": [3],
"values": [0, 1, 2]
},
Here the axes are given as y and x (which are defined correctly earlier in the "domain"
part), however the shape and values do not match that.
I tested the support of dimension name change in the NcML module in Hyrax 1.15.4 (I am not sure if anything has changed in 1.16 - thanks if so!). It worked well until I noticed this error message, showed up when a changed dimension name was referred to in an element that attempted to define the values of a new variable. Defining new variable is one of the functionalities in NcML, and was a working functionality in the previous releases of the module. So I'd like to report it as a backward compatibility issue.
"Hyrax - Bad Request (400)
NCMLModule ParseError: at *.ncml line=881: Product of dimension sizes exceeds the maximum DAP2 size of 2147483647 (2^31-1)!"
I can send you the offending .ncml file if this is indeed a new problem.
Source code is
if (install_dir.empty() && !install_dir.empty()) {
Suggest code rework.
@rrimer07, @hemphilc, @RiverHendriksen and I are working on implementing a CovJSON module for serving CovJSON responses through a Hyrax deployment.
We will be submitting a PR for a feature branch in the short term, it will link to this issue.
In modules/hdf4_handler/HDFEOS2.cc there are many annoying warnings like this:
HDFEOS2.h:675:51: warning: dynamic exception specifications are deprecated in C++11 [-Wdeprecated]
675 | std::vector < Dimension * >&dims) throw (Exception);
| ^~~~~
According to this StackOverflow question, these dynamic exceptions can be simply removed, and they will become an error in C++17, so should be removed:
https://stackoverflow.com/questions/50254641/dynamic-exception-specifications-are-deprecated
I have experimented and removing the throws specification in the function declaration and the header seems to work fine and resolve the warning. I will submit a PR, but would like to see the CI working first...
I tried to visualize data on the browser in tabular form. With DAP2 everything seems to work as expected, but with DAP4 things don't go as planned.
Pasting this url: http://test.opendap.org/opendap/data/nc/sst.mnmean.nc.gz.ascii?sst[3][4][5] on a browser shows:
Dataset: sst.mnmean.nc
sst.lon, 10
sst.sst[sst.time=19813][sst.lat=80], -51
If I paste the dap4 url: http://test.opendap.org/opendap/data/nc/sst.mnmean.nc.gz.dap.csv?dap4.ce=/sst[3][4][5] onto a browser it downloads the sst.mnmean.nc.gz.csv file. Looking at its contents with a simple text editor
Dataset: sst.mnmean.nc
/sst[0][0],
With pandas:
That is pretty confusing.
If I set the url instead as http://test.opendap.org/opendap/data/nc/sst.mnmean.nc.gz.dap.csv?dap4.ce=/sst[3:4][4:5][5:6], it downloads a csv with the sst.mnmean.nc.gz.csv file with values now:
Dataset: sst.mnmean.nc
/sst[0][0], -51, -123
/sst[0][1], 175, 143
/sst[1][0], 50, -18
/sst[1][1], 242, 206
With DAP4 I was expected to see the browser data in tabular form as in DAP2 case. Instead:
a) Downloads data rather than visualize on browser.
b) When inspecting the contents of the csv
file, the data seems incorrect.
Check for getdap and python in configure.ac. Both are required for make check to succeed.
I am working on parsing dmrpp metadata files to match the format of zarr metadata files (parser PR here). The only piece of missing information I have encountered in dmrpp files is the zlib (deflate) compression level. It seems like the opendap dmrpp generator has this logic to retrieve and store this attribute as deflateLevel
(see example below) but it is not implemented across many files. Many NASA datasets including SWOT L2 SSH
, MUR L4 SST
, and Ice-SAT
are missing this field.
When doing a make -j check, I see this warning:
extT.cc: In member function ‘int extT::check_extensions(int, std::map<std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char> >&, std::map<std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char> >&)’:
extT.cc:106:5: warning: no return statement in function returning non-void [-Wreturn-type]
106 | }
| ^
And the test fails.
Looking in the code, I see that the int function is returning without a value. I added a "return 0" and now the test works.
Remove the CmrContainer from the CMR Module (or make it an option).
When I use the suggested procedure to build hyrax and dependencies from https://github.com/OPENDAP/hyrax
(source spath.sh; ./hyrax_clone.sh; ./hyrax_build.sh),
libncml_module.so fails its checks and is not being installed.
My guess is that it is because of a missing "rpath" to the icu library, which was built from source as a dependency and was not installed in a standard location. At least running "ldd" on libncml_module.so shows the missing dependency. Manually copying libncml_module.so to build/lib/bes and adding build/deps/lib to LD_LIBRARY_PATH makes the ncml module work.
Trying to build 3.20.11 for Fedora rawhide with hdf5-1.12-1:
/bin/sh ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I../.. -I gctp/include -I. -I../../dispatch -I../../dap -I/usr/include/libdap -I/usr/include/libxml2 -DMODULE_NAME=\"hdf5_handler\" -DMODULE_VERSION=\"1.16.6\" -I/usr/include/cfitsio -I/usr/include/tirpc -I/include -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection --std=c++11 -c -o HDF5Array.lo HDF5Array.cc
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../.. -I gctp/include -I. -I../../dispatch -I../../dap -I/usr/include/libdap -I/usr/include/libxml2 -DMODULE_NAME=\"hdf5_handler\" -DMODULE_VERSION=\"1.16.6\" -I/usr/include/cfitsio -I/usr/include/tirpc -I/include -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection --std=c++11 -c HDF5Array.cc -fPIC -DPIC -o .libs/HDF5Array.o
HDF5Array.cc: In member function 'bool HDF5Array::m_array_of_reference_new_h5_apis(hid_t, hid_t)':
HDF5Array.cc:840:84: error: request for member 'data' in 'rbuf', which is of pointer type 'H5R_ref_t*' (maybe you meant to use '->' ?)
840 | if(H5Dread(dset_id,H5T_STD_REF,mem_space_id,file_space_id,H5P_DEFAULT,rbuf.data())<0)
| ^~~~
HDF5Array.cc:849:67: error: request for member 'data' in 'rbuf', which is of pointer type 'H5R_ref_t*' (maybe you meant to use '->' ?)
849 | H5R_type_t ref_type = H5Rget_type((const H5R_ref_t *)rbuf.data());
| ^~~~
make[5]: *** [Makefile:862: HDF5Array.lo] Error 1
In [1], you can see
DATA_GRANULE_PR_CAL_COEF {
Float32 Vdata_field_transCoef -0.5199999809;
For the above attribute in [2], you will get
<Attribute name="DATA_GRANULE_PR_CAL_COEF" type="Container">
<Attribute name="Vdata_field_transCoef" type="Byte">
<Value>-0.5199999809</Value>
</Attribute>
The type should be "Float32", not "Byte".
[1] https://eosdap.hdfgroup.org:8080/opendap/data/NASAFILES/hdf4/1B21_CSI.990906.10217.KORA.6.HDF.das
Building without gdal, I see that configure.ac correctly determined that gdal was not present:
configure: Looking for GDAL.
checking for gdal-config... no
configure: WARNING: gdal-config does not exist or it is not an exectuable file
no
However, the build then breaks:
DapFunctions.cc:29:10: fatal error: gdal.h: No such file or directory
29 | #include <gdal.h> // needed for scale_{grid,array}
| ^~~~~~~~
compilation terminated.
This is because DapFunctions includes gdal.h in an unprotected way:
#include "config.h"
#include <iostream>
#include <gdal.h> // needed for scale_{grid,array}
#include <ServerFunctionsList.h>
I tried wrapping this in ifdefs for HAVE_GDAL, and that helps, but eventually we fail like this:
In file included from DapFunctions.cc:55:
ScaleGrid.h:72:13: error: ‘GDAL_GCP’ was not declared in this scope
72 | std::vector<GDAL_GCP> get_gcp_data(libdap::Array *x, libdap::Array *y, int sample_x = 1, int sample_y = 1);
| ^~~~~~~~
ScaleGrid.h:72:21: error: template argument 1 is invalid
72 | std::vector<GDAL_GCP> get_gcp_data(libdap::Array *x, libdap::Array *y, int sample_x = 1, int sample_y = 1);
| ^
ScaleGrid.h:72:21: error: template argument 2 is invalid
So either more code needs to be protected, or gdal needs to be required, and configure.ac should error out if gdal is not found.
Similarly, if HDF5 is not found, the build proceeds, then errors like this:
make[5]: Entering directory '/home/ed/bes/modules/dmrpp_module'
g++ -DHAVE_CONFIG_H -I. -I../.. -I../.. -I../../dispatch -I../../dap -I../../http -I/usr/include/libdap -I/usr/include/libxml2 -I/usr/local/opt/openssl/include -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"1.1.2\" -I./../hdf5_handler -I/include -Wno-vla-extension -g -O2 --pedantic --std=c++11 -MT build_dmrpp-build_dmrpp.o -MD -MP -MF .deps/build_dmrpp-build_dmrpp.Tpo -c -o build_dmrpp-build_dmrpp.o `test -f 'build_dmrpp.cc' || echo './'`build_dmrpp.cc
build_dmrpp.cc:40:10: fatal error: H5Ppublic.h: No such file or directory
40 | #include <H5Ppublic.h>
| ^~~~~~~~~~~~~
compilation terminated.
ma
Hello, do you have a plan to develop a module to read Zarr files with N5?
I've started looking at the code, but I can't see if the work required seems huge or not. If I want to try to develop it, which module should I use as an example? NetCDF? Thanks for your help.
update the Hyrax server functions so that they use the new STAREmaster sidecar files
Please add a way to specify an address (in particular, "127.0.0.1") to which bes's server port should bind.
cf ServerApp::run()
in server/ServerApp.cc
I get this when I do:
autoreconf -i && CXXFLAGS="-fpermissive -Wno-deprecated" ./configure --with-hdf4=/usr/local/hdf-4.2.15_fPIC --with-hdf5=/usr/local/hdf5-1.10.7 --with-hdfeos2=/usr/local/hdfeos_fPIC --prefix=/usr/local/bes-3.20.7 --enable-developer && make -j && sudo make install && sudo make -j check
This is on the branch that includes the --enable-developer fix (#444 )
## ----------------------------------------------- ##
## bes 3.20.7 test suite: bes/dap/tests testsuite. ##
## ----------------------------------------------- ##
1: BESCMD $abs_srcdir/bescmd/agg_array_dds_one_ce.bescmd ok
2: BESCMD $abs_srcdir/bescmd/agg_array_dods_one_ce.bescmd FAILED (testsuite.at:8)
3: BESCMD $abs_srcdir/bescmd/function_result_unwrap_dds.bescmd ok
4: BESCMD $abs_srcdir/bescmd/function_result_unwrap_dods.bescmd FAILED (testsuite.at:12)
5: BESCMD $abs_srcdir/bescmd/function_result_unwrap_dap.bescmd expected failure (testsuite.at:13)
6: BESCMD $abs_srcdir/bescmd/dap4_ce_function-01.bescmd expected failure (testsuite.at:15)
7: BESCMD $abs_srcdir/bescmd/fnoc.das.bescmd ok
8: BESCMD $abs_srcdir/bescmd/fnoc.ddx.bescmd ok
9: BESCMD $abs_srcdir/bescmd/fnoc.dmr.bescmd ok
## ------------- ##
## Test results. ##
## ------------- ##
ERROR: All 9 tests were run,
4 failed (2 expected failures).
## -------------------------- ##
## testsuite.log was created. ##
## -------------------------- ##
Please send `dap/tests/testsuite.log' and all information you think might help:
To: <[email protected]>
Subject: [bes 3.20.7] testsuite: 2 4 failed
You may investigate any problem if you feel able to do so, in which
case the test suite provides a good starting point. Its output may
be found below `dap/tests/testsuite.dir'.
Please change "fro" to "from"
These data files come fro a variety of sources. Some from THG and other were generated
using the mkChunkedTestData.py python script located in the parent directory.
make check fails like this. Apparently make check is trying to do an install?
make[3]: Entering directory '/home/ed/bes/cmdln/tests'
/bin/mkdir -p '/usr/local/bes-3.20.7/share/mds'
/bin/mkdir: cannot create directory ‘/usr/local/bes-3.20.7’: Permission denied
According to the specification:
https://covjson.org/spec/#ndarray-objects
an NdArray shall be one-dimensional.
This is not true in the current covjson response in Hyrax. For example:
shows a 2-D array for range "ColumnAmountO3", although the JSON files shows it is an NdArray type.
At the moment (as of 1.13.3) using server-side functions can result in modified variable and attribute names. For instance the following CDL snippet after running roi() with a fileout netcdf response. This breaks the other-wise CF-compliance of data coming out of BES. Prepending strings like "row_subset." is not desired.
float roi_subset.Latitude(FakeDim0, nXtrack) ;
roi_subset.Latitude:_FillValue = -1.267651e+30f ;
roi_subset.Latitude:missing_value = -1.267651e+30f ;
roi_subset.Latitude:scale_factor = 1. ;
roi_subset.Latitude:add_offset = 0. ;
roi_subset.Latitude:title = "Latitude of the center of the groundpixel" ;
roi_subset.Latitude:units = "degrees_north" ;
roi_subset.Latitude:UniqueFieldDefinition = "Aura-Shared" ;
roi_subset.Latitude:origname = "Latitude" ;
roi_subset.Latitude:fullnamepath = "/HDFEOS/SWATHS/ColumnAmountNO2/Geolocation Fields/Latitude" ;
I have a .nc file with a variable with an attribute missing_value = NaNf
. When I download this file from opendap in netcdf3 or netcdf4 format I get missing_value = 0.0f
in the resulting file.
I expect to get the same NaNf
as in the original file
Here's a script to reproduce the issue (requires bash, docker, wget and ncdump)
https://gist.github.com/samoylovfp/ba1df8ef41f27c44ddeed9f2d4076fc9
I suspect that istringstream
is the culprit in these places
https://github.com/OPENDAP/bes/blob/6724e44/modules/fileout_netcdf/FONcAttributes.cc#L389
https://github.com/OPENDAP/bes/blob/6724e44/modules/fileout_netcdf/FONcAttributes.cc#L1030
Because it cannot parse NaN:
https://ideone.com/bhTntI
https://stackoverflow.com/questions/11420263/is-it-possible-to-read-infinity-or-nan-values-using-input-streams
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.