pmbio / limix-backup Goto Github PK

Have tried all possible methods for install (pip, ubuntu install, and from source) all fail because of a missing file in the cython/lmm_forest directory: SplittingCore.c:

building 'limix.ensemble.SplittingCore' extension
creating build/temp.linux-x86_64-2.7/tmp/pip-build-nuSWha/limix/cython
creating build/temp.linux-x86_64-2.7/tmp/pip-build-nuSWha/limix/cython/lmm_forest
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -I/tmp/pi

p-build-nuSWha/limix/External -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/p
ython2.7 -c /tmp/pip-build-nuSWha/limix/cython/lmm_forest/SplittingCore.c -o build/temp.linux-x86_64
-2.7/tmp/pip-build-nuSWha/limix/cython/lmm_forest/SplittingCore.o -Wno-comment -Wno-unused-but-set-v
ariable -Wno-overloaded-virtual -Wno-uninitialized -Wno-delete-non-virtual-dtor -Wunused-variable -s
td=c++11
x86_64-linux-gnu-gcc: error: /tmp/pip-build-nuSWha/limix/cython/lmm_forest/SplittingCore.c: No s
uch file or directory
x86_64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 4

----------------------------------------

Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-nuSWha/limix/se
tup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), _file
_, 'exec'))" install --record /tmp/pip-LLSca2-record/install-record.txt --single-version-externally-
managed --compile" failed with error code 1 in /tmp/pip-build-nuSWha/limix/

Hi

Iam having a problem following the 'loading files into LIMIX' tutorial, specifically with using limix_converter to convert the phenotype_sample.csv into hdf5 format.

This is the command Iam using :

limix_converter --outfile=./my_pheno.hdf5 -csv=./phenotype_sample.csv

It seems to run ok, but the hdf5 file does not contain any datasets or groups.

h5dump -A my_pheno.hdf5 
HDF5 "my_pheno.hdf5" {
GROUP "/" {
}
}

I am using this on a biolinux box
Linux XXX 3.13.0-68-generic #111-Ubuntu SMP Fri Nov 6 18:17:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

and using LIMIX version limix==0.7.12 installed using pip.

I have also tried using my own phenotype file (.csv) but have the same problem.
Can anyone shed any light?
Cheers
Alex

Hi,
I am looking for some help to get genotype/phenotype data into the HDF5 format required by LIMIX. do you have some example scripts or can you point me to the relevant parts of the code?

thanks in advance,
Inti

Hi
There are a couple of deps that the Limix master of today (13/08/2014) has that are not mentioned in the docs:

• h5py
• pandas

The deps are checked in setup.py but not mentioned anywhere else. H5PY is a dep for the limix_runner binary that is part of the Python bindings.

I got above problem in windows python.
cl: Command line error D8021: invalid numeric argument '/Wno-comment'
errror: command 'C:\User\Administrator\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe' failed with exit status 2

I am using windows 7, and already have installed the latest version of "anaconda".
I am trying to install your package via "pip install limix", the first error, I got was about visual studio, so, I downloaded visual studio for python (VCForPython27.msi), installed it, and added it to the environment variable "PATH".

Now I am getting a different error :

C:\Users\ndv680\AppData\Local\Continuum\Anaconda>pip install limix
Collecting limix
Using cached limix-0.7.12.tar.gz
Requirement already satisfied (use --upgrade to upgrade): Cython in c:\users\ndv680\appdata\local\continuum\anaconda\lib\site-packag
es (from limix)
Requirement already satisfied (use --upgrade to upgrade): h5py in c:\users\ndv680\appdata\local\continuum\anaconda\lib\site-packages
(from limix)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.6.1 in c:\users\ndv680\appdata\local\continuum\anaconda\lib\site-
packages (from h5py->limix)
Requirement already satisfied (use --upgrade to upgrade): six in c:\users\ndv680\appdata\local\continuum\anaconda\lib\site-packages
(from h5py->limix)
Building wheels for collected packages: limix
Running setup.py bdist_wheel for limix
Complete output from command C:\Users\ndv680\AppData\Local\Continuum\Anaconda\python.exe -c "import setuptools;file='c:\users
\ndv680\appdata\local\temp\pip-build-mjzxp0\limix\setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), _file
_, 'exec'))" bdist_wheel -d c:\users\ndv680\appdata\local\temp\tmpb4vkblpip-wheel-:
running bdist_wheel
running build
running build_ext
building 'deprecated._core' extension
creating build
creating build\temp.win-amd64-2.7
creating build\temp.win-amd64-2.7\Release
creating build\temp.win-amd64-2.7\Release\src
creating build\temp.win-amd64-2.7\Release\src\interfaces
creating build\temp.win-amd64-2.7\Release\src\interfaces\python
creating build\temp.win-amd64-2.7\Release\src\limix
creating build\temp.win-amd64-2.7\Release\src\limix\covar
creating build\temp.win-amd64-2.7\Release\src\limix\gp
creating build\temp.win-amd64-2.7\Release\src\limix\io
creating build\temp.win-amd64-2.7\Release\src\limix\likelihood
creating build\temp.win-amd64-2.7\Release\src\limix\LMM
creating build\temp.win-amd64-2.7\Release\src\limix\LMM_forest
creating build\temp.win-amd64-2.7\Release\src\limix\mean
creating build\temp.win-amd64-2.7\Release\src\limix\modules
creating build\temp.win-amd64-2.7\Release\src\limix\utils
creating build\temp.win-amd64-2.7\Release\External
creating build\temp.win-amd64-2.7\Release\External\nlopt
creating build\temp.win-amd64-2.7\Release\External\nlopt\direct
creating build\temp.win-amd64-2.7\Release\External\nlopt\cdirect
creating build\temp.win-amd64-2.7\Release\External\nlopt\praxis
creating build\temp.win-amd64-2.7\Release\External\nlopt\luksan
creating build\temp.win-amd64-2.7\Release\External\nlopt\crs
creating build\temp.win-amd64-2.7\Release\External\nlopt\mlsl
creating build\temp.win-amd64-2.7\Release\External\nlopt\mma
creating build\temp.win-amd64-2.7\Release\External\nlopt\cobyla
creating build\temp.win-amd64-2.7\Release\External\nlopt\newuoa
creating build\temp.win-amd64-2.7\Release\External\nlopt\neldermead
creating build\temp.win-amd64-2.7\Release\External\nlopt\auglag
C:\Users\ndv680\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS-
/DNDEBUG -Isrc -IExternal -IExternal/nlopt -I./External/nlopt/stogo -I./External/nlopt/util -I./External/nlopt/direct -I./External/n
lopt/cdirect -I./External/nlopt/praxis -I./External/nlopt/luksan -I./External/nlopt/crs -I./External/nlopt/mlsl -I./External/nlopt/m
ma -I./External/nlopt/cobyla -I./External/nlopt/newuoa -I./External/nlopt/neldermead -I./External/nlopt/auglag -I./External/nlopt/bo
byqa -I./External/nlopt/isres -I./External/nlopt/slsqp -I./External/nlopt/api -I./External/nlopt/esch -IC:\Users\ndv680\AppData\Loca
l\Continuum\Anaconda\lib\site-packages\numpy\core\include -IC:\Users\ndv680\AppData\Local\Continuum\Anaconda\include -IC:\Users\ndv6
80\AppData\Local\Continuum\Anaconda\PC /Tp./src\interfaces\python\limix_wrap.cpp /Fobuild\temp.win-amd64-2.7\Release./src\interface
s\python\limix_wrap.obj -Wno-comment -Wno-unused-but-set-variable -Wno-overloaded-virtual -Wno-uninitialized -Wno-unused-const-varia
ble -Wno-unknown-warning-option -Wno-shorten-64-to-32
cl : Command line error D8021 : invalid numeric argument '/Wno-comment'
error: command 'C:\Users\ndv680\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe
' failed with exit status 2

Failed building wheel for limix
Failed to build limix
Installing collected packages: limix
Running setup.py install for limix
Complete output from command C:\Users\ndv680\AppData\Local\Continuum\Anaconda\python.exe -c "import setuptools, tokenize;file_
_='c:\users\ndv680\appdata\local\temp\pip-build-mjzxp0\limix\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file
).read().replace('\r\n', '\n'), file, 'exec'))" install --record c:\users\ndv680\appdata\local\temp\pip-ybs1p4-record\install-re
cord.txt --single-version-externally-managed --compile:
running install
running build
running build_ext
building 'deprecated._core' extension
C:\Users\ndv680\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS

• /DNDEBUG -Isrc -IExternal -IExternal/nlopt -I./External/nlopt/stogo -I./External/nlopt/util -I./External/nlopt/direct -I./External
  /nlopt/cdirect -I./External/nlopt/praxis -I./External/nlopt/luksan -I./External/nlopt/crs -I./External/nlopt/mlsl -I./External/nlopt
  /mma -I./External/nlopt/cobyla -I./External/nlopt/newuoa -I./External/nlopt/neldermead -I./External/nlopt/auglag -I./External/nlopt/
  bobyqa -I./External/nlopt/isres -I./External/nlopt/slsqp -I./External/nlopt/api -I./External/nlopt/esch -IC:\Users\ndv680\AppData\Lo
  cal\Continuum\Anaconda\lib\site-packages\numpy\core\include -IC:\Users\ndv680\AppData\Local\Continuum\Anaconda\include -IC:\Users\nd
  v680\AppData\Local\Continuum\Anaconda\PC /Tp./src\interfaces\python\limix_wrap.cpp /Fobuild\temp.win-amd64-2.7\Release./src\interfa
  ces\python\limix_wrap.obj -Wno-comment -Wno-unused-but-set-variable -Wno-overloaded-virtual -Wno-uninitialized -Wno-unused-const-var
  iable -Wno-unknown-warning-option -Wno-shorten-64-to-32
  cl : Command line error D8021 : invalid numeric argument '/Wno-comment'
  error: command 'C:\Users\ndv680\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.e
  xe' failed with exit status 2

my googling result http://stackoverflow.com/questions/13717551/1cl-command-line-error-d8021-invalid-numeric-argument-wno-deprecated-er
I appreciate to hear your comments on that.

Not sure if this is the function I should be using as it is placed in the deprecated folder.

I get the following when trying to call genotype_reader_h5py.getPos:
NameError: global name 'snp_idx' is not defined

Just commented out this line and it worked

elif snp_idx is not None:    
    return self.position.iloc[snp_idx]

Guessing this was useful at getting out 1 particular SNP by index.

When trying to run python setup.py with --prefix=foo the --prefix string is passed to SCons which doesn't work as SCons returns with:
SCons Error: no such option: --prefix.

It might be helpful for users if LIMIX raised a warning when/if the user provides a "covariance matrix" that is not semi-definite positive.

Currently if this happens in test_lmm, the user just gets an array of NAs.

import scipy as sp
import scipy.linalg as la
import warnings

## read in / generate covariance_matrix

eigenvals, eigenvecs = la.eigh(covariance_matrix)
min_eigenval = eigenvals.min()
if min_eigenval < 0: warnings.warn('The covariance is not semi-definite positive (min eigenv=%.2e<0)'%min_eigenval)

I tried to install limix from source. I have installed the dependencies manually (pandas, h5py, swig). Then the installation of limix seemed to work without problems. When I tried to import the package, I got an error message about a missing module 'mtset'. Therefore I installed the mtSet module from the git repository (https://github.com/PMBio/mtSet.git), but I still get the same error message:

[sfux@euler02 ~]$ module load gcc/4.8.2 gdc python/3.4.3 swig/3.0.8 hdf5/1.8.12          
Autoloading openblas/0.2.13_seq
[sfux@euler02 ~]$ python
Python 3.4.3 (default, Mar  3 2016, 14:52:31) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import limix
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/cluster/apps/gdc/python/3.4.3/lib64/python3.4/site-packages/limix-0.7.74-py3.4-linux-x86_64.egg/limix/__init__.py", line 2, in <module>
    from .mtSet import MTSet
  File "/cluster/apps/gdc/python/3.4.3/lib64/python3.4/site-packages/limix-0.7.74-py3.4-linux-x86_64.egg/limix/mtSet/__init__.py", line 1, in <module>
    from mtset import MTSet
ImportError: No module named 'mtset'

I was a bit surprised, because importing 'mtSet' works without giving an error message:

>>> import mtSet
>>> 

How can I get limix running ? All the required modules are installed. Is it a typo in the limix module that it tries to import 'mtset' instead of 'mtSet' ?

Thank you very much for your help

Hi

PROBLEM1
I am having a problem following the 'loading files into LIMIX' tutorial, specifically with using limix_converter to convert my phenotype file 'phenotypes.csv into hdf5 format.

If I use:

limix_converter -O ./my_file.hdf5 -C ./phenotypes.csv

I obtain something like that:

/home/jon/anaconda2/lib/python2.7/site-packages/limix/io/conversion.py:78: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support sep=None with delim_whitespace=False; you can avoid this warning by specifying engine='python'.
  C = pandas.io.parsers.read_csv(csv_file,sep=sep,header=None,index_col=False,*args,**kw_args)

On the other hand, if I use:

limix_converter -O ./my_file.hdf5 -C ./phenotypes.csv -D ,

in order to avoid this warning, I obtain:

......,21478,21479,21480,21481,21482,21483,21484,21485,21486,21487,21488,21489,21490,21491,21492,21493,21494,21495,21496,21497,21498,21499,21500,21501,21502,21503,21504,21505,21506,21507,21508,21509,21510,21511,21512,21513,21514,21515,21516,21517,21518,21519,21520,21521,21522,21523,21524,21525,21526,21527,21528,21529,21530,21531,21532,21533,21534,21535,21536,21537,21538,21539,21540,21541,21542,21543,21544,21545,21546,21547,21548,21549,21550,21551,21552,21553,21554,21555,21556,21557,21558,21559,21560,21561,21562,21563,21564,21565,21566,21567,21568,21569,21570,21571,21572,21573,21574,21575,21576,21577,21578,21579,21580,21581,21582,21583,21584,21585,21586,21587,21588,21589,21590,21591,21592,21593,21594,21595,21596,21597,21598,21599,21600,21601,21602,21603,21604,21605,21606,21607,21608,21609,21610,21611,21612,21613,21614,21615,21616,21617,21618,21619,21620,21621,21622,21623,21624,21625,21626,21627,21628,21629,21630,21631,21632,21633,21634,21635,21636,21637,21638,21639,21640,21641,21642,21643,21644,21645,21646,21647,21648,21649,21650,21651,21652,21653,21654,21655,21656,21657,21658,21659,21660,21661,21662,21663,21664,21665,21666,21667,21668,21669,21670,21671,21672,21673,21674,21675,21676,21677,21678,21679,21680,21681,21682,21683,21684,21685,21686,21687,21688,21689,21690,21691,21692,21693,21694) have mixed types. Specify dtype option on import or set low_memory=False.

In both cases, it seems to convert to hdf5 correctly. However, I do not if this last warning is due to the massive table of phenotypes I am using (I have around 21000 genes and 184 samples).

PROBLEM2
Anyway, once I made the conversion of the file, I do:

geno_reader  = gr.genotype_reader_tables('my_file.hdf5')
pheno_reader = phr.pheno_reader_tables('my_file.hdf5')
dataset = data.QTLData(geno_reader=geno_reader,pheno_reader=pheno_reader)

If I look at:

pheno_reader.pheno_matrix
pheno_reader.sample_ID
pheno_reader.phenotype_ID

Everything seems to be OK. However, when I do:

phenotypes,sample_idx=pheno_reader.getPhenotypes()

I have the following warning:

/home/jon/anaconda2/lib/python2.7/site-packages/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)
/home/jon/anaconda2/lib/python2.7/site-packages/numpy/core/_methods.py:82: RuntimeWarning: Degrees of freedom <= 0 for slice
  warnings.warn("Degrees of freedom <= 0 for slice", RuntimeWarning)

And 'phenotypes' is an empty dataframe, and only the column names are defined ([0 rows x 21694 columns]). I do not understand why, but it also happens to me when I use your sample file of phenotypes.

Many thanks,
Jon

Among other failures with python-2.7 and enthought environment I see things like

stvjc@aura ~/LIMIX $ pip install limix
Downloading/unpacking limix
Downloading limix-0.7.12.tar.gz (3.0MB): 3.0MB downloaded
Running setup.py (path:/tmp/pip_build_stvjc/limix/setup.py) egg_info for package limix
Traceback (most recent call last):
File "", line 17, in
File "/tmp/pip_build_stvjc/limix/setup.py", line 104, in
install_distributions(['Cython'])
File "/tmp/pip_build_stvjc/limix/setup.py", line 90, in install_distributions
command = pip.commands.install.InstallCommand()
AttributeError: 'dict' object has no attribute 'install'
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

File "", line 17, in

File "/tmp/pip_build_stvjc/limix/setup.py", line 104, in

install_distributions(['Cython'])

File "/tmp/pip_build_stvjc/limix/setup.py", line 90, in install_distributions

command = pip.commands.install.InstallCommand()

AttributeError: 'dict' object has no attribute 'install'

Let me know if there are more details needed, but if you will just indicate what kind
of linux or macosx infrastructure will permit installation that would be fine.

I think there is a bug in the significance calculations when the user provides qvalues. These changes might fix it.

if thr==None:
thr = 0.01
if qv == None: thr = 0.01/float(posCum.shape[0])

if qv==None:
Isign = pv<thr
else:
Isign = qv<thr # I think this is ment to be qv not pv

pypi shows limix package as hcache and the description is shown as UNKNOWN.

Potentially related to #4 . I recently installed limix from source; it's importable and usable, but does not appear either in pip freeze, or by manually listing libraries using pip.

>>> import os
>>> os.system('cat /proc/version')
Linux version 2.6.32-431.20.3.el6.x86_64 (mockbuild@c6b9.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Thu Jun 19 21:14:45 UTC 2014
0
>>> import pip
>>> import limix
>>> print('\n'.join(sorted(['{}=={}'.format(i.key, i.version) for i in pip.get_installed_distributions()])))
biopython==1.64
blist==1.3.6
brewer2mpl==1.4.1
cython==0.21
fastlmm==0.2.2
guppy==0.1.10
h5py==2.4.0b1
matplotlib==1.4.2
mmh3==2.3
mock==1.0.1
nose==1.3.4
numpy==1.9.0
pandas==0.14.1
patsy==0.3.0
pyfasta==0.5.2
pyparsing==2.0.3
pysam==0.8.0
python-dateutil==2.2
pytz==2014.7
pyvcf==0.6.7
scikit-learn==0.15.2
scipy==0.14.0
scons==2.3.4
six==1.8.0
statsmodels==0.6.1
virtualenv==12.0.dev1
wsgiref==0.1.2

Hi,
When loading data. Limix complains when the chromosomes codes are not numeric, for example scaffold_338

Any chance to relax that?

Thanks in advance

When using limix_converter to convert my phenotype.csv file to hdf5 format the sample_ID 's have a decimal '.0' appended to them.

Here are a few rows of my phenotype file with the first column the numeric sample ID's

,F_grain_Mo,F_grain_Zn,A_FT,A_PH
1,1.1233,18.58,81,128.3
3,0.7292,16.79,83,133.8
4,1.215,17.01,93,147.7
5,1.1612,21.26,108,174.8
6,0.8666,16.08,101,171.3

I convert this csv file to hdf5 format using

limix_converter -O ./phenotype.hdf5 -C ./phenotype.csv -D ,

but the sample_ID dataset in the phenotype group are

1.0
3.0
4.0
5.0
6.0

This is likely to cause me problems when subsetting the genotype and phenotype hdf5 files for individual samples. Is there any way around this behaviour?
Cheers
Alex

Here's the relevant output from pip install limix:

    no such option: --egg-base

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 2 in /private/var/folders/5b/7lpc81ss0nbbk9l2x956zs5m0000gq/T/pip-build-Kj0cUP/limix