Code Monkey home page Code Monkey logo

spt3g_software's Introduction

About

This repository contains the public portions of the analysis software developed by the SPT-3G collaboration. It includes the IO libraries, data acquisition code for McGill's DfMux readout boards, pipeline glue code, a binned map-maker, and a sky mock-observer, as well as defining all the APIs used by the project. For more detail on the intended processing architecture, please see the Quick Start chapter of the documentation.

Except where otherwise noted, all files are distributed under the 2-clause BSD license. Acknowledgments and patches are appreciated.

Documentation

The main documentation for the software is in the docs folder. After building the software, you can build a pretty, searchable copy by running make docs.

Dependencies

This depends on Boost and cmake, as well as the usual Python packages. Some additional packages (NetCDF, in particular) will activate optional components of the code if installed. You also need a C++11 compiler. This software is designed to run and work on a variety of operating systems (all Linuxes, Mac OS X, and FreeBSD) and architectures (at least 64-bit x86 and POWER).

Minimum versions:

  • GCC >= 5.0 or clang >= 3.4
  • Boost >= 1.48
  • cmake >= 3.5
  • Python >= 2.7 (although pre-Python-3 support is best-effort)

On Ubuntu/Debian, you can install the non-Python dependencies, including the optional ones, by doing:

apt-get install cmake libboost-all-dev libflac-dev libnetcdf-dev

On RHEL-type systems (SL, CentOS, etc.), do this:

yum install cmake netcdf-devel boost-devel flac-devel

If your system defaults to Python 2, but you wish to use Python 3, please do the following:

  1. Install Python 3 from the system package manager
  2. Make sure the python-3 version of the Boost library is installed (on Ubuntu, this is part of the standard boost-python package referenced above)
  3. When you run cmake below, pass -DPython_EXECUTABLE=`which python3`

On any system, this software requires numpy and scipy (hard requirements), plus astropy and healpy (optional).

Setup on RHEL6

Note that on any RHEL6 system, you will need a newer compiler than ships with the OS. Please follow whatever directions apply at your site to achieve this. Alternately, if you have OASIS set up on your local system, run this before anything else (also works on several other operating systems, including RHEL7):

eval `/cvmfs/spt.opensciencegrid.org/py3-v4/setup.sh`

How to Build

To build:

cd spt3g_software
mkdir build
cd build
cmake ..
make

To build the documentation in the build directory type:

./env-shell.sh make docs

This will construct an html version of the documentation. This builds the documentation in the build/docs folder. Open build/docs/index.html in your favorite web browser. You should at least read the quick start portion of the documentation before getting started.

Installation

For various reasons it may be useful to install the software after building, instead of continuing to use it out of the build directory. Several CMake variables control how the software is installed:

  • WITH_BZIP2, which defaults to TRUE, is used to control whether the core library is built with support for bzip2 compression of G3 files. Use -DWITH_BZIP2=FALSE when calling cmake to disable.
  • CMAKE_INSTALL_PREFIX, which defaults to /usr/local is used as the root directory for installing all non-python components (header files, cmake export scripts, etc.)
  • PYTHON_MODULE_DIR, which if not explicitly set defaults to the result of running distutils.sysconfig.get_python_lib <https://docs.python.org/3/distutils/apiref.html#distutils.sysconfig.get_python_lib> with the selected python interpreter, is where the python module will be installed.

It is rarely necessary to set PYTHON_MODULE_DIR if python has been detected correctly, but setting CMAKE_INSTALL_PREFIX is frequently useful when installing into a python virtual environment. In such a case, one may want build as follows:

cd spt3g_software
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX="${VIRTUAL_ENV}"
make
make install

After this completes, it should be possible when using the virtual environment to import spt3g in python without needing to make use of env-shell.sh.

Release Version Tracking

Use git tags to keep track of release versions. Tags should be of the form "v0.1.2" for release with major version 0, minor version 1 and patch version 2. If such a tag is defined, cmake will populate the following outputs:

  • A cmake/Spt3gConfigVersion.cmake file that contains the version number to be checked when including the Spt3g libraries in another cmake project
  • A spt3g/version.py file containing VCS parameters for access in python and stored in PipelineInfo frames
  • Add a SPT3G_VERSION compiler definition for accessing the version string in C++ code

When exporting the source tree to a standalone archive, run the following command in the source directory to ensure that the source version is correctly exported:

cmake/config_export.sh

Then archive the source tree using git archive as usual.

Version Control Hygiene

You can use two mechanisms to access the repository: git and SVN. The following is a brief overview of how to use these in a way that your collaborators will appreciate.

Git

To initially check out the repository:

git clone https://[email protected]/CMB-S4/spt3g_software.git

To update your checkout (the --rebase is important, especially if you have local changes):

git pull --rebase

To send your changes back:

git diff files_to_commit <- Examine this
git commit files_to_commit
git push

SVN

To initially check out the repository:

svn co https://[email protected]/CMB-S4/spt3g_software/trunk spt3g_software

To update your checkout:

svn up

To send your changes back:

svn diff files_to_commit <- Examine this
svn ci files_to_commit

spt3g_software's People

Contributors

adamanderson avatar arahlin avatar cailmdaley avatar cnweaver avatar cozzyd avatar dpdutcher avatar ghallsimpsons avatar jlashner avatar johngroh avatar joshuamontgomery avatar keskitalo avatar kferguson42 avatar kimmywu avatar marius311 avatar menanteau avatar mhasself avatar ndhuang avatar nlharr avatar nwhitehorn avatar panzhaodi avatar pchich avatar samtomguns avatar tcrawfor avatar tnatoli avatar tskisner avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

spt3g_software's Issues

Cannot use empty G3TimestreamMap() constructor

Building with #95 breaks on the private SouthPoleTelescope/spt3g_software repo:

[ 79%] Building CXX object todfilter/CMakeFiles/todfilter.dir/src/fftutils.cpp.o
/github/actions-runner/_work/spt3g_software/spt3g_software/todfilter/src/fftutils.cpp: In function ‘void fft_filter_mem_friendly(const G3TimestreamMap&, const G3VectorComplexDouble&, G3TimestreamMap&, bool, G3MapDoubleConstPtr)’:
/github/actions-runner/_work/spt3g_software/spt3g_software/todfilter/src/fftutils.cpp:307:30: error: use of deleted function ‘G3TimestreamMap& G3TimestreamMap::operator=(const G3TimestreamMap&)’
   out_data = G3TimestreamMap();
                              ^
In file included from /github/actions-runner/_work/spt3g_software/spt3g_software/todfilter/include/todfilter/numericutils.h:9,
                 from /github/actions-runner/_work/spt3g_software/spt3g_software/todfilter/src/fftutils.cpp:3:
/github/actions-runner/_work/spt3g_software/spt3g_software/core/include/core/G3Timestream.h:176:7: note: ‘G3TimestreamMap& G3TimestreamMap::operator=(const G3TimestreamMap&)’ is implicitly declared as deleted because ‘G3TimestreamMap’ declares a move constructor or move assignment operator
 class G3TimestreamMap : public G3FrameObject,
       ^~~~~~~~~~~~~~~
/github/actions-runner/_work/spt3g_software/spt3g_software/todfilter/src/fftutils.cpp: In function ‘void fft_filter_mem_friendly_w_multi_filters(G3TimestreamMap&, G3VectorDouble&, G3MapVectorComplexDouble&, G3TimestreamMap&)’:
/github/actions-runner/_work/spt3g_software/spt3g_software/todfilter/src/fftutils.cpp:433:43: error: use of deleted function ‘G3TimestreamMap& G3TimestreamMap::operator=(const G3TimestreamMap&)’
         filtered_ts_map = G3TimestreamMap();
                                           ^
At global scope:
cc1plus: error: unrecognized command line option ‘-Wno-self-assign-overloaded’ [-Werror]
cc1plus: error: unrecognized command line option ‘-Wno-constant-conversion’ [-Werror]
cc1plus: error: unrecognized command line option ‘-Wno-deprecated-register’ [-Werror]
cc1plus: error: unrecognized command line option ‘-Wno-unknown-warning-option’ [-Werror]
cc1plus: all warnings being treated as errors
gmake[2]: *** [todfilter/CMakeFiles/todfilter.dir/build.make:115: todfilter/CMakeFiles/todfilter.dir/src/fftutils.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:10[88](https://github.com/SouthPoleTelescope/spt3g_software/actions/runs/4684229589/jobs/8300157535#step:7:89): todfilter/CMakeFiles/todfilter.dir/all] Error 2
gmake: *** [Makefile:141: all] Error 2
Error: Process completed with exit code 2.

Sequential package versions

It would be very useful to have sequential version codes available through the python interface. In so3g we will often need to require / check for some minimum spt3g version that implements a feature or bug-fix.

I know it's onerous to be constantly tagging and releasing. If the commit-differential tags like "v0.1.2+23-g7ab341" are available, somehow, then that's sufficient to bridge even large gaps between releases (for my purposes).

Mapuitls Not Found

@arahlin I just tried to build spt3g on a new computer and it looks like we're missing src/maputils.cpp. Did it not get added to your recent commit?

-- Found GSL: /usr/lib/x86_64-linux-gnu/libgsl.so;/usr/lib/x86_64-linux-gnu/libgslcblas.so  
-- Found FFTW_THREADS: /usr/lib/x86_64-linux-gnu/libfftw3_threads.so  
-- Found FFTW: /usr/lib/x86_64-linux-gnu/libfftw3.so  
-- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDES) 
-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found version "3.1") 
-- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")  
-- + calibration
-- + coordinateutils
-- + core
-- Found Flac: /usr/lib/x86_64-linux-gnu/libFLAC.so  
-- + dfmux
-- Found NetCDF: /usr/lib/x86_64-linux-gnu/libnetcdf.so  
-- + gcp
-- Configuring done
CMake Error at coordinateutils/CMakeLists.txt:7 (add_library):
  Cannot find source file:

    src/maputils.cpp

  Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp
  .hxx .in .txx


CMake Error: CMake can not determine linker language for target: coordinateutils
CMake Error: Cannot determine link language for target "coordinateutils".
-- Generating done
-- Build files have been written to: /home/mcmlab/software/spt3g_software/build

Sparse backend for healpix masks

Consider storing boolean masks for healpix maps in a sparse data structure to save memory with maps with small sky coverage (<10%).

G3Timestream Slicing

Slicing of G3Timetreams in python differs a bit from np.array slicing.

In python you are allowed to slice lists with stop and step being larger than the size of the array:

>>> import numpy as np
>>> x = np.arange(5.)
>>> x[:10]
array([0., 1., 2., 3., 4.])
>>> x[::10]
array([0.])

However G3Timestreams produce fatal errors when trying the same thing:

>>> from spt3g import core
>>> import numpy as np
>>> x = core.G3Timestream(np.arange(5.))
>>> x[:10]
FATAL (G3Timestream): Stop index 10 out of range (G3Timestream.cxx:822 in G3TimestreamPtr {anonymous}::G3Timestream_getslice(const G3Timestream&, boost::python::slice))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/so1/shared/software/spt3g_software/200627/build/spt3g/core/timestreamextensions.py", line 13, in G3Timestream_getitem
    it = x._cxxslice(y)
RuntimeError: Stop index 10 out of range (in G3TimestreamPtr {anonymous}::G3Timestream_getslice(const G3Timestream&, boost::python::slice))
>>> x[::10]
FATAL (G3Timestream): Step index 10 out of range (G3Timestream.cxx:824 in G3TimestreamPtr {anonymous}::G3Timestream_getslice(const G3Timestream&, boost::python::slice))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/so1/shared/software/spt3g_software/200627/build/spt3g/core/timestreamextensions.py", line 13, in G3Timestream_getitem
    it = x._cxxslice(y)
RuntimeError: Step index 10 out of range (in G3TimestreamPtr {anonymous}::G3Timestream_getslice(const G3Timestream&, boost::python::slice))

It would be great if we could change G3Timestream so slicing behavior is consistent with numpy arrays.

G3Reader fails on multiple files when last file is empty

Create empty file:

In [3]: wr = core.G3Writer('empty.g3')

In [4]: del wr

Create non-empty file:

In [5]: fr = core.G3Frame()

In [6]: fr['A'] = 0

In [7]: wr = core.G3Writer('full.g3')

In [8]: wr(fr)
Out[8]: [<spt3g.core.G3Frame at 0x7fd5034ac450>]

In [9]: del wr

Read empty file only:

In [10]: p = core.G3Pipeline()

In [11]: p.Add(core.G3Reader, filename = ['empty.g3'])

In [12]: p.Run()  #runs fine, no errors

Read full file first and then empty file:

In [13]: p = core.G3Pipeline()

In [14]: p.Add(core.G3Reader, filename = ['full.g3', 'empty.g3'])

In [15]: p.Run()
ERROR (G3Reader) 18-Jan-2024:11:11:52 CST: Exception raised while reading file empty.g3 (G3Reader.cxx:98 in virtual void G3Reader::Process(G3FramePtr, std::deque<boost::shared_ptr<G3Frame> >&))
WARN (Unknown) 18-Jan-2024:11:11:52 CST: Exception in module "spt3g.core.G3Reader" (N6cereal9ExceptionE): Failed to read 1 bytes from input stream! Read 0 (G3Pipeline.cxx:119 in size_t {anonymous}::PushFrameThroughQueue(G3FramePtr, bool, bool, rusage&, std::vector<{anonymous}::G3Pipeline_mod_data>&, std::vector<{anonymous}::G3Pipeline_mod_data>::iterator, int&, std::deque<{anonymous}::G3Pipeline_proc_data>&, G3FramePtr&))
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-ecb398338890> in <module>
----> 1 p.Run()

RuntimeError: Failed to read 1 bytes from input stream! Read 0

Note: other combinations (read full file only, read empty file first and full file second) also work fine.

Python G3VectorTime(G3VectorTime &) constructor is broken?

I suspect the recent buffs for buffer protocol may have broken G3VectorTime constructor. See below.

>>> from spt3g import core
>>> import numpy as np
>>> t0 = core.G3VectorTime(np.array([100000000, 200000000]))
>>> t0
spt3g.core.G3VectorTime([01-Jan-1970:00:00:01.000000000, 01-Jan-1970:00:00:02.000000000])
>>> t1 = core.G3VectorTime(t0)
>>> t1
spt3g.core.G3VectorTime([01-Jan-1970:00:00:01.000000000, 01-Jan-1970:00:00:02.000000000, 01-Jan-1970:00:00:00.000005290, 11-Jan-1970:21:29:27.377759360])
>>> len(t0), len(t1)
(2, 4)

Expected result: t1 should be a copy of t0.

Couple other observations:

  • This seems to impact Python only ... similar code in C++ works as expected.
  • It does not matter how you construct the G3VectorTime -- if I construct it from a single G3Time instead of a numpy array, the same doubling occurs.

G3VectorQuat from buffer doesn't catch odd strides

When creating G3VectorQuat from numpy array, it seems to check for shape (-1, 4) but does not handle non-trivial strides.

Consider test program:

from spt3g import core
import numpy as np

quats = np.array([[1., 2., 3., 4., 5.],   # a                                                           
                  [0., 0., 0., 0., 0.],   # b                                                           
                  [0., 0., 0., 0., 0.],   # c                                                           
                  [0., 0., 0., 0., 0.]])  # d                                                           

for test in [
    quats,             # bad shape
    quats[:,:4],       # ok shape, non-trivial strides
    quats.T,           # ok shape, non-trivial strides
    quats.T.copy(),    # ok shape, trivial strides
    ]:
    try:
        print(repr(core.G3VectorQuat(test)))
    except TypeError:
        print('(TypeError)')

Produces output (on version 0.3-71-g613d0d4):

(TypeError)
(TypeError)
spt3g.core.G3VectorQuat([(1,2,3,4), (5,0,0,0), (0,0,0,0), (0,0,0,0), (0,0,0,0)])
spt3g.core.G3VectorQuat([(1,0,0,0), (2,0,0,0), (3,0,0,0), (4,0,0,0), (5,0,0,0)])

But expected output is:

(TypeError)
spt3g.core.G3VectorQuat([(1,2,3,4), (0,0,0,0), (0,0,0,0), (0,0,0,0)])
spt3g.core.G3VectorQuat([(1,0,0,0), (2,0,0,0), (3,0,0,0), (4,0,0,0), (5,0,0,0)])
spt3g.core.G3VectorQuat([(1,0,0,0), (2,0,0,0), (3,0,0,0), (4,0,0,0), (5,0,0,0)])

OR

(TypeError)
(TypeError)
(TypeError)
spt3g.core.G3VectorQuat([(1,0,0,0), (2,0,0,0), (3,0,0,0), (4,0,0,0), (5,0,0,0)])

(In the latter case, the code would simply rejecting anything that doesn't have simple C ordering.)

Arithmetic broken for G3 numeric types

G3 numeric types (and their vector equivalents) all seem to have some broken arithmetic operations, either with numpy and python scalars, or even with themselves. Here is a sampling of things that raise TypeError: unsupported operand...:

G3Int + G3Int
int + G3Int
G3Int + int
G3VectorInt + int
G3VectorDouble + G3VectorDouble
float * G3VectorDouble
float + G3VectorDouble
G3VectorDouble + float
G3VectorComplexDouble * float

Notably, the follwing do work:

np.float64 * G3VectorDouble
G3VectorDouble * np.float64
G3VectorDouble * float
G3VectorComplexDouble * float64

Build system fails to find boost_python3 on OSX

CMake fails to find boost_python on my system, even though it seems to be looking in the right directory. I'm running OSX with Macports-installed Boost.

Here is output from CMake:

...
-- Found PythonInterp: /opt/local/bin/python (found version "3.7.2") 
-- Found PythonLibs: /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7m.dylib (found version "3.7.2") 
CMake Error at /opt/local/share/cmake-3.13/Modules/FindBoost.cmake:2100 (message):
  Unable to find the requested Boost libraries.

  Boost version: 1.66.0

  Boost include path: /opt/local/include

  Could not find the following Boost libraries:

          boost_python

  Some (but not all) of the required Boost libraries were found.  You may
  need to install these additional Boost libraries.  Alternatively, set
  BOOST_LIBRARYDIR to the directory containing Boost libraries or BOOST_ROOT
  to the location of Boost.
Call Stack (most recent call first):
  CMakeLists.txt:58 (find_package)


-- Boost Geometry Available
-- Could NOT find GSL (missing: GSL_LIBRARIES GSL_INCLUDES) 
-- Found FFTW_THREADS: /opt/local/lib/libfftw3_threads.dylib 
...

The library in question exists at /opt/local/lib/libboost_python3-mt.dylib and it looks like there is code in CMakeLists.txt to handle this case: https://github.com/CMB-S4/spt3g_software/blob/master/CMakeLists.txt#L42-L57 but it is looking for libpython3.*\\.so to get the version number. Replacing that glob with libpython3.*\\.dylib fixes the issue on OSX-systems but breaks it for everyone else.

Broken export of boost linking flags

Hello @arahlin , just a heads up that the recent change to Spt3gBoostPython.cmake:

set(CMAKE_FIND_PACKAGE_PREFER_CONFIG TRUE)

has broken the ability for the so3g package to get boost linking information by using the build/cmake-install/Spt3gConfig.cmake file. Can you give more information about what problem that was trying to fix? I have not yet dug into this any deeper to see why the various Boost::filesystem, Boost::python, etc are no longer available, but thought I would try to get more information about this change first.

Batch conversion of timestamps

Is there a fast way to convert between arrays of G3VectorTime and numpy float arrays? I find myself writing this a lot:

  g3times = core.G3VectorTime([core.G3Time(t * core.G3Units.s) for t in timestamps])

and that's no fun. I am hoping for something like:

  g3times = core.G3VectorTime(timestamps * core.G3Units.s)
  timestamps = np.array(g3times) / core.G3Units.s

Build issue with gcc >= 11

Got errors like the ones shown below. On gcc version 12.2.0, boost 1.74.

Had to add -Wno-mismatched-new-delete flag. Happy to PR -- I don't know how safe it is to just add -Wno flags for everyone whenever things like this happen. Probably this is transient due to older boost / newer gcc.

In file included from /home/mhasse/code/public/anaconda3/envs/py38/include/boost/checked_delete.hpp:15,
                 from /home/mhasse/code/public/anaconda3/envs/py38/include/boost/smart_ptr/detail/sp_counted_impl.hpp:27,
                 from /home/mhasse/code/public/anaconda3/envs/py38/include/boost/smart_ptr/detail/shared_count.hpp:27,
                 from /home/mhasse/code/public/anaconda3/envs/py38/include/boost/smart_ptr/shared_ptr.hpp:17:
In function ‘void boost::checked_delete(T*) [with T = float]’,
    inlined from ‘boost::detail::shared_count::shared_count(Y*) [with Y = float]’ at /home/mhasse/code/public/anaconda3/envs/py38/include/boost/smart_ptr/detail/shared_count.hpp:154:34,
    inlined from ‘void boost::detail::sp_pointer_construct(boost::shared_ptr<X>*, Y*, shared_count&) [with T = float; Y = float]’ at /home/mhasse/code/public/anaconda3/envs/py38/include/boost/smart_ptr/shared_ptr.hpp:279:20,
    inlined from ‘boost::shared_ptr<T>::shared_ptr(Y*) [with Y = float; T = float]’ at /home/mhasse/code/public/anaconda3/envs/py38/include/boost/smart_ptr/shared_ptr.hpp:373:44,
    inlined from ‘static G3TimestreamPtr G3Timestream::G3TimestreamPythonHelpers::timestream_from_iterable(boost::python::api::object, G3Timestream::TimestreamUnits)’ at /home/mhasse/work/code/simonso/spt3g_software/core/src/G3Timestream.cxx:929:31:
/home/mhasse/code/public/anaconda3/envs/py38/include/boost/core/checked_delete.hpp:36:5: error: ‘void operator delete(void*, std::size_t)’ called on pointer returned from a mismatched allocation function [-Werror=mismatched-new-delete]
   36 |     delete x;
      |     ^~~~~~~~

Incorrect behavior of `numpy.abs()` on `G3VectorQuat` objects

In [1]: from spt3g.core import quat, G3VectorQuat
   ...: import numpy as np
   ...: q = quat(1.0, 0.0, 0.0, 0.0)
   ...: v = G3VectorQuat([q, q, q, q])

In [2]: abs(v)  # the builtin function has the correct behavior
Out[2]: spt3g.core.G3VectorDouble([1, 1, 1, 1])

In [3]: np.abs(v)  # expect this to return a 1-d numpy array
Out[3]:
array([[1., 0., 0., 0.],
       [1., 0., 0., 0.],
       [1., 0., 0., 0.],
       [1., 0., 0., 0.]])

Segfault on buffers with non-trivial strides

Instantiating G3Timestream from a numpy array works, but only if the buffer is close-packed in memory. Passing a view into an array, where strides are not 1, yields a segfault.

import spt3g
from spt3g import core
import numpy as np

print(spt3g.version.gitrevision)

for n in [1, 2, -2]:
    print(n)
    data = np.zeros(100, 'float64')[::n]
    print(core.G3Timestream(data))

Results in:

c45570b13da0cdaaab0b519344892d5053f99b20
1
100 samples at inf Hz
2
Segmentation fault (core dumped)

Expected result: the non-trivial stride should either quietly work, or raise an exception.

Imp module is deprecated

We should find a replacement for imp.load_dynamic in spt3g.core.load_pybindings. The imp module will be removed in python 3.12.

G3SkyMapMask is not efficiently stored on disk

G3SkyMap mask objects, which are essentially std::vector<bool> objects with some extra features, are packed on disk as one byte per element, rather than 1 bit per element as one would expect. This is rather inefficient, especially when masks are used as input arguments to pipeline modules (and therefore end up in memory-intensive G3PipelineInfo frames all over the place).

C++11 mode and -Werror prevent building with new Boost versions

In file included from /Users/cweaver/Work/CMB-S4/spt3g_software-public-git/core/src/G3Vector.cxx:3:
In file included from /Users/cweaver/Work/CMB-S4/spt3g_software-public-git/core/include/core/container_pybindings.h:5:
In file included from /Users/cweaver/Work/CMB-S4/spt3g_software-public-git/core/include/core/G3Map.h:6:
In file included from /Users/cweaver/Work/CMB-S4/spt3g_software-public-git/core/include/core/G3Quat.h:6:
In file included from /usr/local/include/boost/math/quaternion.hpp:14:
/usr/local/include/boost/math/tools/config.hpp:23:6: error: "The minimum language standard to use Boost.Math will be C++14 starting in July 2023 (Boost 1.82 release)" [-Werror,-W#warnings]
#    warning "The minimum language standard to use Boost.Math will be C++14 starting in July 2023 (Boost 1.82 release)"
     ^
1 error generated.

The simplest solution is to change the language mode to C++14, but I would like to first verify that this will be available on all platforms we consider supported.

Logging timestamps enabled by default

I find timestamps on the core logging messages very useful for rudimentary profiling, especially for long running scripts. While the profile option for G3Pipeline.Run() provides similar functionality, it's only useful for scripts that are entirely pipeline-based, which is often not the case. Moreover, timestamps on log messages make it easy to track the progress of a script in real-time, rather than waiting until it has finished running to find out how long it took.

I would like to propose enabling timestamps by default for the G3PrintfLogger class, or at the very least making it easier to turn that feature on quickly (e.g. with a root level function that doesn't require remembering where in the depths of the logging code that attribute is set).

Ellipsis for repr of vector

numpy repr of long arrays is abbreviated:

>>> numpy.zeros(10000)
array([0, 0, 0, ..., 0, 0, 0])

But G3Vector types just print every single element, making it easy to flood an interactive session with pages of unwanted output. Is there any objection to implementing a similar abbreviation in vec_repr? Since repr does not use Description(), the latter could continue to present a complete dump of the vector, in case that's needed.

GZIP_2 compression of a flat-sky map seems lossy in some cases

When we save a flat-sky map in a FITS file via maps.fitsio.save_skymap_fits, the default option for the argument compress is 'GZIP_2'. This compression scheme is supposed to be lossless, but that does not seem to be the case when I tested the compression scheme on an SPT-3G winter field coadd.

Here is an example:

In[4]: frame = core.G3File(path_to_a_coadd).next()


# Default save

In [5]: maps.fitsio.save_skymap_fits(output_path, frame["T"], Q=frame["Q"], U=frame["U"])

In [6]: frame_debug = maps.fitsio.load_skymap_fits(output_path)

In [7]: numpy.allclose(frame["T"], frame_debug["T"])
Out[7]: False


# No-compression save

In [18]: maps.fitsio.save_skymap_fits(output_path, frame["T"], Q=frame["Q"], U=frame["U"], compress=False, overwrite=True)

In [19]: frame_debug = maps.fitsio.load_skymap_fits(output_path)

In [20]: numpy.allclose(frame["T"], frame_debug["T"])
Out[20]: True

Boost 1.82+ requires standard C++14

Compiling with the latest version of boost now produces the following warning:

warning "The minimum language standard to use Boost.Math will be C++14 starting in July 2023 (Boost 1.82 release)

We should update the source to comply for the time being. In the long run, we'll want to put more effort into migrating away from Boost.

Installation target?

It's often convenient to install packages rather than running them from the build directory. Can we get Makefile targets to help with that?

Using an installed Python package is one thing -- building against installed lib/include is another. Probably the installed Python package is the most immediately useful.

Inconsistent numpy detection

I'm finding that numpy is not detected when cmake is run a second time after the initial setup. This happens on my mac, but I don't think the issue is OS-specific.

The first time cmake is run I get this:

$ cmake ..
...
-- Found Python: /usr/local/Frameworks/Python.framework/Versions/3.11/bin/python3.11 (found version "3.11.4") found components: Interpreter Development Development.Module Development.Embed
-- Found NumPy: /usr/local/lib/python3.11/site-packages/numpy/core/include
-- Found SciPy: FOUND
-- Python modules will be installed to /usr/local/lib/python3.11/site-packages
...

The second time I get this:

$ cmake ..
-- Found NumPy: NOT FOUND
-- Found SciPy: FOUND
-- Python modules will be installed to /usr/local/lib/python3.11/site-packages
...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.