cvxgrp / cvxportfolio Goto Github PK

TypeErrorTraceback (most recent call last)
in ()
8 all_return_estimates[(t,tp1)]=return_estimate.loc[tp1]
9
---> 10 returns_forecast = cp.MPOReturnsForecast(all_return_estimates)
11 results_MPO={}

TypeError: Can't instantiate abstract class MPOReturnsForecast with abstract methods weight_expr

In my experience, when a portfolio becomes infeasible, Policy.get_trades "defaults to no-trades" (usually because it has become close to zero value, and the single leverage limit constraint turns infeasible), the logger built into the Result classes calls

    def log_policy(self, t, exec_time):
        self.log_data("policy_time", t, exec_time)
        # TODO mpo policy requires changes in the optimization_log methods
        if not isinstance(self.policy, MultiPeriodOpt):
            for cost in self.policy.costs:
                self.log_data("policy_" + cost.__class__.__name__,
                                  t, cost.optimization_log(t))

However, the various Costs (HcostModel at least), if the problem is infeasible, will have a self.expression that is None. Thus the following throws a NoneType has no attribute 'A1' error.

    def optimization_log(self, t):
        # BAD: A1 MAY NOT EXIST IF PROBLEM INFEASIBLE
        return self.expression.value.A1

I have modified log_policy so that if this occurs, the value that is logged is simply np.NaN. Otherwise, the entire simulation halts.

What is A1, and what value should be logged if Cost.expression.value is None?

Also, sometimes the portfolio value drops below zero. For instance, losses on short positions are not bounded. In such cases, this assertion in MultiPeriodOpt fails:

class MultiPeriodOpt(SinglePeriodOpt):
        def get_trades(self, portfolio, t=pd.datetime.today()):
            value = sum(portfolio)
            assert (value > 0.) # This assumption can actually break.
            #...

What should occur here?

Also, when the value becomes negative, the nonlinear "second_term" produces a complex number and breaks the optimization.

class TcostModel(BaseCost):

    def _estimate(self, t, w_plus, z, value):
        # omitting lots of code
        second_term = time_locator(self.nonlin_coeff, t) * \
            time_locator(self.sigma, t) * \
            (value / time_locator(self.volume, t))**(self.power - 1) 
        # If value becomes negative, this produces a complex number.

And then when the portfolio becomes negative, it continues trading.

Maybe there is a simpler solution that retrofitting all the code to accept negative value?

Thanks,
Joey

P.S. I've been a long-time lurker to this project

SinglePeriodOptimization:

RealTimeOptimization and Hello world

MultiPeriodOptimization

In line 75 of returns.py there is a call to self.weight_expr(t, wplus). This should be self.weight_expr(tau, wplus).

When running the MPO example it fails when trying to access the data loaded into risk_data:

In[3]: risk_data.exposures
Traceback (most recent call last): File "/home/xxx/envs/cvxportfolio/lib/python3.6/site-packages/pandas/io/pytables.py", line 1423, in _create_storer return globals()[_STORER_MAP[pt]](self, group, **kwargs) KeyError: 'wide'

I assume this is due to a wrong version of pandas. I am using pandas 0.25.1 and python 3.6, but read in thread #44 that some people use pandas 0.20.3. Could someone provide a complete list of the packages compatible with cvxportfolio?

When running MultiPeriodOptimization.ipynb, getting the error:

Can't instantiate abstract class MPOReturnsForecast with abstract methods weight_expr

SimulationResult object contains policy and simulator objects. Both take a lot of spaces.
This is causing problem if we run parameter tuning codes because many SimulationResult object will be created and co-exist in memory. I suggest we add in an option to return concise version of SimulationResult object(remove policy and simulator objects and keep only the relevant simulation metrics).
Here is what I can think of:

In run_backtest and run_multiple_backtest function in policies.py
run_backtest(....., return_concise_results=False, ...)
.....
if return_concise_results:
results = results.concise_version
return results

run_multiple_back_test(..., return_concise_results=False, ...)

Then in SimulationResult.py
@Property
def concise_version(self):
self.policy=None
self.simulator = None
return self

I am trying to follow the paper to set up factor neutral soft constraint as objective cost in Multi-Period-Optimization. From the resulting trade schedule, you can see that for the first trade, factor neutral is achieved. However for the rest of period, factor is far from neutral.
Anyone has any insight on this?

The setup I have look like the following:
`class FactorNeutralSoftConstraintCost (BaseCost):
def init (self, factor_exposure, priority):
self.priority=priority
self.factor_exposure = factor_exposure
super(FactorNeutralSoftConstraintCost, self).init()

       def _estimate(self, t, w_plus, z, value):
              w_plus = w_plus[:-1]  # remove cash
              self.expression = time_matrix_locator(factor_exposure, t) * w_plus                   
               return cvx.norm(cvx.multiply(self.priority, self.expression), 1)`

When attempting to upgrade cvxportfolio to 0.0.5 using the instructions, I ran into the following conflicts:

(base) 14:24:48 thisguy@mypc thisdir $ conda install -c cvxgrp cvxportfolio --dry-run
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - cvxportfolio
  - enum34
Use "conda info <package>" to see the dependencies for each package.

I am using python 2.7 and had recently already done an update to anaconda. Do you have any suggestions or do I have to do a complete reinstall?

Hi,

I am trying to run an mpo example and would like to use the FullSigma risk model. I am passing a prediction vector for each period created like all_return_estimates in the MultiPeriodOptimization example. I would like to pass an associated full sigma for each period. I am currently doing this using the same dictionary format as the return estimates. This does not seem to work. Is it possible to pass multiple sigmas, one for each period? If so, what format should be used? (In the code comments, there seems to be a reference to the obsolete pd.Panel format)

Thanks.

hello @enzobusseti @SteveDiamond @jonathanng

thank you for providing this great framework for portfolio optimization. I am wondering would you mind spent a little time fixing some minor bugs in source code and in examples/tests.

Some bugs likes:
(1) Can't instantiate abstract class MPOReturnsForecast with abstract methods weight_expr.

(2) ValueError: Incompatible dimensions (430, 1) (431, 1)

(3) parameter name mistake, like returns_forecast and alpha_model.

(4) the first argument of SPO policy in SPO examples.

thanks for your time and have a good day.

When I ran the MultipPeriodOPtimization example, the volumes is loaded from csv.gz file and passed as a dataframe to the TCostModel setup.

volumes=pd.read_csv(datadir+'volumes.csv.gz',index_col=0,parse_dates=[0]).iloc[:,:-1]
....
...
simulated_tcost = cp.TcostModel(half_spread=0.0005/2., nonlin_coeff=1., sigma=sigmas, volume=volumes)

The error happens on the following line:
def null_checker(obj):
"""Check if obj contains NaN."""
if (isinstance(obj, pd.Panel) or
isinstance(obj, pd.DataFrame) or
isinstance(obj, pd.Series)):
if np.any(pd.isnull(obj)): # this line spit error
raise ValueError('Data object contains NaN values', obj)
elif np.isscalar(obj):
if np.isnan(obj):
raise ValueError('Data object contains NaN values', obj)
else:
raise TypeError('Data object can only be scalar or Pandas.')
---- I got the following error:
alueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

by changing it to np.any(pd.isnull(obj)).any() seems to fix the problem

Hi all,

While it seems there was a small push about a year ago to update the project, there are far too many broken issues (benchmarks don't work, for example). Should I assume this project is dead/abandoned?

Cheers

I suggest to add in support for time-varying factor exposure support in FactorMaxLimit and FactorMinLimit constraints.
By doing so, user can pass in factor exposures as Pandas panel data with time as item index. Then in the _weight_expr(self, t, w_plus, z, v) function. call time_matrix_locator(exposure, t) to extract the exposure data. Then everything would flow through.
Also, the factor limit can be pass in as a dataframe with time as index to support time varying limit as well.

This way, these two constraints would support both SinglePeriodOpt and MultiPeriodOpt.

I have this error when I run the HelloWorld example.
Can anybody tell me how to fix it? Thank you.

market_sim=cp.MarketSimulator(returns, [tcost_model, hcost_model], cash_key='USDOLLAR') 
init_portfolio = pd.Series(index=returns.columns, data=0.)
init_portfolio.USDOLLAR = 100
result = market_sim.run_backtest(init_portfolio,
                               start_time='2013-01-03',  end_time='2016-12-31',  
                               policy=spo_policy)

result.summary()
--------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-9-b229f1c1f6b4> in <module>()
      4 result = market_sim.run_backtest(init_portfolio,
      5                                start_time='2013-01-03',  end_time='2016-12-31',
----> 6                                policy=spo_policy)
      7 
      8 result.summary()

/home/weiwu/projects/cvxportfolio/cvxportfolio/simulator.py in run_backtest(self, initial_portfolio, start_time, end_time, policy, loglevel)
    114             start = time.time()
    115             try:
--> 116                 u = policy.get_trades(h, t)
    117             except cvx.SolverError:
    118                 logging.warning(

/home/weiwu/projects/cvxportfolio/cvxportfolio/policies.py in get_trades(self, portfolio, t)
    222 
    223         for cost in self.costs:
--> 224             cost_expr, const_expr = cost.weight_expr(t, wplus, z, value)
    225             costs.append(cost_expr)
    226             constraints += const_expr

/home/weiwu/projects/cvxportfolio/cvxportfolio/risks.py in weight_expr(self, t, w_plus, z, value)
     43 
     44     def weight_expr(self, t, w_plus, z, value):
---> 45         self.expression = self._estimate(t, w_plus - self.w_bench, z, value)
     46         return self.gamma * self.expression, []
     47 

/home/weiwu/projects/cvxportfolio/cvxportfolio/risks.py in _estimate(self, t, wplus, z, value)
     81     def _estimate(self, t, wplus, z, value):
     82         try:
---> 83             self.expression = cvx.quad_form(wplus, locator(self.Sigma, t))
     84         except TypeError:
     85             self.expression = cvx.quad_form(

/home/weiwu/projects/cvxportfolio/cvxportfolio/risks.py in locator(obj, t)
     30     """Picks last element before t."""
     31     try:
---> 32         return obj.iloc[obj.axes[0].get_loc(t, method='pad')]
     33     except AttributeError:  # obj not pandas
     34         return obj

/home/weiwu/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/pandas/core/indexes/multi.py in get_loc(self, key, method)
   1946         """
   1947         if method is not None:
-> 1948             raise NotImplementedError('only the default get_loc method is '
   1949                                       'currently supported for MultiIndex')
   1950 

NotImplementedError: only the default get_loc method is currently supported for MultiIndex

I have a question. If we use MPO in liquidation context with alpha estimation. Say
if my initial position is: long symbol A and short symbol B, and I have some short-term alpha estimate.
I want o liquidate this portfolio without buying extra A or selling B in the interim period. Basically I want the trade in A going monotonic --- keep selling A, and the trade in B going monotonic --- keeping buying B, in each of the interim period until I fully liquidate the portfolio.
How can I set up constraint?

Hi,

When I run your MultiPeriodOptimization example, in the first part of MPO Coarse search, I get the error:
TypeError: init() got an unexpected keyword argument 'alpha_model'. Then, I delete "alpha_model =" characters and keep everything else unchanged. However, I still get error saying
init() got multiple values for argument 'trading_times'.

Could you tell me how I can fix it?

I'm wondering if it's perhaps a result of new pandas behaviour, but I had to change the null_checker (line 29 in data_management.py) to read as:

if np.any( np.any(np.any(pd.isnull(obj))) ):

for the HelloWorld example to run. Seems to be an issue in that it spits out a series of "False" values and it requires a single bool value. Can anyone confirm? I'm on Pandas 0.20.3.

cvxportfolio code needs to be upgraded for cvxpy 1.0.3 (current default in PyPi). Current examples and notebook fail as expected. Any plans to upgrade and test?

I have tried all methods I know to install library, it usually comes with error or just running without result

I successfully ran your Jupyter notebook examples using the daily Quandl data and was impressed with the speed; however, when I insert e.g., minutely data, the speed per minute with minutely data is much, much slower than the speed per day with daily data. The difference is ca x20 times larger. How can this be rectified?

Correct me if I am wrong - but I felt there is a bug in the share calculation in this example. It makes more sense to use t-1 close price to calculate the shares instead of t (as shown in the example), in other words - prices.shift(1). Since your decision price is last close - and in the propagation the assumption is filled at previous close and propagated by close to close return.

it looks like the MPO example is using the future return data, since it's impossible to know the exact return estimate for time t+1 at time t. This may make the comparison between SPO and MPO unfair.

I am getting this error

TypeError: cannot properly create the storer for: [_STORER_MAP] [group->/exposures (Group) '',value-><class 
'NoneType'>,format->None

while reading the file risk_model.py.

Is it a known problem?

Hi !
Really love the framework and planning to get involved.
was wanting to expand for a masters project.
i am trying to run the hello world program and getting the following -

risk_model = cp.FullSigma(Sigma_hat)
gamma_risk, gamma_trade, gamma_hold = 5., 1., 1.
leverage_limit = cp.LeverageLimit(3)

spo_policy = cp.SinglePeriodOpt(return_forecast=r_hat,
costs=[gamma_riskrisk_model, gamma_tradetcost_model, gamma_holdhcost_model],
constraints=[leverage_limit])
risk_model = cp.FullSigma(Sigma_hat)
gamma_risk, gamma_trade, gamma_hold = 5., 1., 1.
leverage_limit = cp.LeverageLimit(3)
​
spo_policy = cp.SinglePeriodOpt(return_forecast=r_hat,
costs=[gamma_riskrisk_model, gamma_tradetcost_model, gamma_holdhcost_model],
constraints=[leverage_limit])

ValueError Traceback (most recent call last)
in ()
5 spo_policy = cp.SinglePeriodOpt(return_forecast=r_hat,
6 costs=[gamma_riskrisk_model, gamma_tradetcost_model, gamma_hold*hcost_model],
----> 7 constraints=[leverage_limit])

~/cvxportfolio/cvxportfolio/policies.py in init(self, return_forecast, costs, constraints, solver, solver_opts)
217
218 if not isinstance(return_forecast, BaseReturnsModel):
--> 219 null_checker(return_forecast)
220 self.return_forecast = return_forecast
221

~/cvxportfolio/cvxportfolio/utils/data_management.py in null_checker(obj)
27 isinstance(obj, pd.DataFrame) or
28 isinstance(obj, pd.Series)):
---> 29 if np.any(pd.isnull(obj)):
30 raise ValueError('Data object contains NaN values', obj)
31 elif np.isscalar(obj):

~/tutorial-env2/lib/python3.6/site-packages/pandas/core/generic.py in nonzero(self)
1571 raise ValueError("The truth value of a {0} is ambiguous. "
1572 "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
-> 1573 .format(self.class.name))
1574
1575 bool = nonzero

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

was wondering if you could assist.
Best,
Andrew

for i,t in enumerate(returns.index[:-1]):
all_return_estimates[(t,t)]= return_estimate.loc[t]
tp1=returns.index[i+1]
all_return_estimates[(t,tp1)]=return_estimate.loc[tp1]

returns_forecast = cp.MPOReturnsForecast(all_return_estimates) #<-******* This one returns error
results_MPO={}
---- It returns error:
returns_forecast = cp.MPOReturnsForecast(all_return_estimates)
TypeError: Can't instantiate abstract class MPOReturnsForecast with abstract methods weight_expr

When lookahead_periods > 2, I get the following error:

File "PATH\cvxportfolio\returns.py", line 103, in weight_expr_ahead
    return self.alpha_data[(t, tau)].values.T * wplus
KeyError: (Timestamp('2012-01-03 00:00:00'), Timestamp('2012-01-05 00:00:00'))

Imagine FactorModelSigma is populated with actual T-1 Factor Exposure + Factor Risk information.
In MPO backtest simulation, say if lookahead period = 5, then when generating trading schedule at T=1, you are using the FactorExposure and Factor Risk information of T=2,3,4,5. So the trade schedule you generated has lookahead bias.

The proper way to do the backtest simulation for MPO may need put FactorModelSigma in simulator and for each trading times, get the prevailing FactorExposure and Factor Risk information and use it to generate the MPO trade schedule even though you only trade the first leg and then advance to next trade decision time, do the optimization using new Factor exposure and risk information again.

The definition of long-only constraint in source code includes last cash term, i.e. the weight of cash should be non-negative.

On the other side, I notice there is a inconsistency in the paper with respect to this constraint.

Def 1:
In page 7 line 8, it says "The portfolio is long-only when the asset holdings are all nonnegative, i.e.,(h_t)_i ≥ 0 for i = 1, . . . ,n" This definition ignores the cash, which is n+1 th term of h_t

Def 2:
While in page33, Long only. "This constraint requires that only long asset positions are
held, wt + zt ≥ 0." which requires cash to be nonnegative.

I personally perfer the 1st definition, IMO it is more intuitive and one can combine this long-only constraint with the long-cash constraint to compose the current long-only constraint.

More questions, than anything.

[1] I'm struggling to find a good way to jumpstart a portfolio in a back test simulation. For example, other places may have things like "ignore turnover in first period" and I wondering how you would think about that in this framework.

[2] Max cash constraint so as to make sure the portfolio was fully invested. Or perhaps optimize to a cash target of say, 100bps, for example. Would you suggest I add in a MaxCash constraint similar to the LongCash constraint?

Thanks!

The terminal_weights setup in MultiPeriodOpt example seems to fail when calling run_backtest function.
python 3.6 + cvxportfolio 0.05 + cvxpy 1.0.25 + numpy 1.17.1 + pandas 0.22.0.

terminal_weights = pd.Series(index=returns.columns, data=0)
terminal_weights['USDOLLAR'] = 1

policy = \
      cp.MultiPeriodOpt(return_forecast=returns_forecast,
                        costs=[gamma_risk*risk_model, gamma_tcost*simulated_tcost, optimization_hcost],
                        constraints=[cp.LeverageLimit(3)],
                        trading_times=list(returns.index[(returns.index>=start_t)&(returns.index<=end_t)]),
                        lookahead_periods=lookahead_periods,
                        terminal_weights=terminal_weights)
    
...
result = simulator.run_backtest(w_b*1e8, start_time = start_t, end_time=end_t, policy=policy)

It error out in policies.py file:

if self.terminal_weights is not None:
            prob_arr[-1].constraints += [wplus == self.terminal_weights.values]  # <-- Error Out

The error message is:

File "/home/kec/.local/lib/python3.6/site-packages/cvxportfolio/policies.py", line 374, in get_trades
    prob_arr[-1].constraints += [wplus == self.terminal_weights.values]
AttributeError: can't set attribute

anyone knows how to fix this problem?

run "SinglePeriodOptimization.ipynb" and got the error:
ValueError: Cannot broadcast dimensions (60, 432) (432,)

RemoteTraceback Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
File "/Users/che.guan/opt/anaconda3/lib/python3.7/site-packages/multiprocess/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/Users/che.guan/opt/anaconda3/lib/python3.7/site-packages/multiprocess/pool.py", line 44, in mapstar
return list(map(args))
File "/Users/che.guan/cvxportfolio/cvxportfolio/simulator.py", line 147, in _run_backtest
policy, loglevel=loglevel)
File "/Users/che.guan/cvxportfolio/cvxportfolio/simulator.py", line 116, in run_backtest
u = policy.get_trades(h, t)
File "/Users/che.guan/cvxportfolio/cvxportfolio/policies.py", line 266, in get_trades
cost_expr, const_expr = cost.weight_expr(t, wplus, z, value)
File "/Users/che.guan/cvxportfolio/cvxportfolio/risks.py", line 58, in weight_expr
self.expression = self._estimate(t, w_plus - self.w_bench, z, value)
File "/Users/che.guan/cvxportfolio/cvxportfolio/risks.py", line 159, in _estimate
np.sqrt(values_in_time(self.idiosync, t)), wplus)) +
File "/Users/che.guan/opt/anaconda3/lib/python3.7/site-packages/cvxpy/atoms/affine/binary_operators.py", line 221, in init
super(multiply, self).init(lh_expr, rh_expr)
File "/Users/che.guan/opt/anaconda3/lib/python3.7/site-packages/cvxpy/atoms/affine/binary_operators.py", line 43, in init
super(BinaryOperator, self).init(lh_exp, rh_exp)
File "/Users/che.guan/opt/anaconda3/lib/python3.7/site-packages/cvxpy/atoms/atom.py", line 45, in init
self._shape = self.shape_from_args()
File "/Users/che.guan/opt/anaconda3/lib/python3.7/site-packages/cvxpy/atoms/affine/binary_operators.py", line 258, in shape_from_args
return u.shape.sum_shapes([arg.shape for arg in self.args])
File "/Users/che.guan/opt/anaconda3/lib/python3.7/site-packages/cvxpy/utilities/shape.py", line 49, in sum_shapes
len(shapes)" %s" % tuple(shapes))
ValueError: Cannot broadcast dimensions (60, 432) (432,)
"""

The above exception was the direct cause of the following exception:

ValueError Traceback (most recent call last)
in
2 warnings.filterwarnings('ignore')
3 results.update(dict(zip(policies.keys(), simulator.run_multiple_backtest(1E8*w_b, start_time=start_t,end_time=end_t,
----> 4 policies=policies.values(), parallel=True))))

~/cvxportfolio/cvxportfolio/simulator.py in run_multiple_backtest(self, initial_portf, start_time, end_time, policies, loglevel, parallel)
150 if parallel:
151 workers = multiprocess.Pool(num_workers)
--> 152 results = workers.map(_run_backtest, policies)
153 workers.close()
154 return results

~/opt/anaconda3/lib/python3.7/site-packages/multiprocess/pool.py in map(self, func, iterable, chunksize)
266 in a list that is returned.
267 '''
--> 268 return self._map_async(func, iterable, mapstar, chunksize).get()
269
270 def starmap(self, func, iterable, chunksize=None):

~/opt/anaconda3/lib/python3.7/site-packages/multiprocess/pool.py in get(self, timeout)
655 return self._value
656 else:
--> 657 raise self._value
658
659 def _set(self, i, obj):

ValueError: Cannot broadcast dimensions (60, 432) (432,)

Hello,

I ran the HelloWorld jupyter notebook on both a Linux VM and a Mac (actual Mac, not VM), and got the same error. Pasted below is the code of the failing cell (the 7th cell) followed by the output of the cell:

market_sim=cp.MarketSimulator(returns, [tcost_model, hcost_model], cash_key='USDOLLAR') 
init_portfolio = pd.Series(index=returns.columns, data=250000.)
init_portfolio.USDOLLAR = 0
results = market_sim.run_multiple_backtest(init_portfolio,
                               start_time='2013-01-03',  end_time='2016-12-31',  
                               policies=[spo_policy, cp.Hold()])
results[0].summary()

RemoteTraceback Traceback (most recent call last)

RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/gperraul/anaconda3/lib/python3.7/site-packages/multiprocess/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/home/gperraul/anaconda3/lib/python3.7/site-packages/multiprocess/pool.py", line 44, in mapstar
return list(map(*args))
File "/home/gperraul/PycharmProjects/cvxportfolio/cvxportfolio/simulator.py", line 147, in _run_backtest
policy, loglevel=loglevel)
File "/home/gperraul/PycharmProjects/cvxportfolio/cvxportfolio/simulator.py", line 116, in run_backtest
u = policy.get_trades(h, t)
File "/home/gperraul/PycharmProjects/cvxportfolio/cvxportfolio/policies.py", line 265, in get_trades
cost_expr, const_expr = cost.weight_expr(t, wplus, z, value)
File "/home/gperraul/PycharmProjects/cvxportfolio/cvxportfolio/risks.py", line 58, in weight_expr
self.expression = self._estimate(t, w_plus - self.w_bench, z, value)
File "/home/gperraul/PycharmProjects/cvxportfolio/cvxportfolio/risks.py", line 105, in _estimate
wplus, values_in_time(self.Sigma, t))
File "/home/gperraul/anaconda3/lib/python3.7/site-packages/cvxpy/atoms/quad_form.py", line 215, in quad_form
raise Exception("Invalid dimensions for arguments.")
Exception: Invalid dimensions for arguments.
"""

The above exception was the direct cause of the following exception:

Exception Traceback (most recent call last)

in
4 results = market_sim.run_multiple_backtest(init_portfolio,
5 start_time='2013-01-03', end_time='2016-12-31',
----> 6 policies=[spo_policy, cp.Hold()])
7 results[0].summary()
8

~/PycharmProjects/cvxportfolio/cvxportfolio/simulator.py in run_multiple_backtest(self, initial_portf, start_time, end_time, policies, loglevel, parallel)
150 if parallel:
151 workers = multiprocess.Pool(num_workers)
--> 152 results = workers.map(_run_backtest, policies)
153 workers.close()
154 return results

~/anaconda3/lib/python3.7/site-packages/multiprocess/pool.py in map(self, func, iterable, chunksize)
266 in a list that is returned.
267 '''
--> 268 return self._map_async(func, iterable, mapstar, chunksize).get()
269
270 def starmap(self, func, iterable, chunksize=None):

~/anaconda3/lib/python3.7/site-packages/multiprocess/pool.py in get(self, timeout)
655 return self._value
656 else:
--> 657 raise self._value
658
659 def _set(self, i, obj):

Exception: Invalid dimensions for arguments.

Do you know what might explain this error?

I used the latest Anaconda (2019.10) to install cvxpy in both the linux VM and Mac (I had errors when trying to install cvxpy using pip). Not sure if it could somehow be related to that.. Perhaps I should use an older version of the multiprocess library?

Thanks in advance.

@enzobusseti Quandl is used within HelloWorld.ipynb, RealTimeOptimization.ipynb and DataEstimatesRiskModel.ipynb
Quandl performs communication with a remote server whose internals cannot be inspected and audited (essential properties for financial management software). I proposed you a powerful addon for hyperparameter optimization which consists of about 100 lines of open source code to make API requests to a remote solver (Quandl performs similar requests to its remote server). You classified it as a closed code and mentioned that inserting components with closed source is contrary to the philosophy of this work, make it less attractive to users, and less trustworthy. Therefore, in order to be consistent with your previous assessment, Quandl should be removed.

When running MPO example, I got error message on this line in policy.py file:

z = cvx.Variable(*w.size) #----- Error Out
wplus = w + z

It looks like the line does not work for any cvxpy version > cvxpy1.0. Need to change it to
z = cvx.Variable(w.size) to make it work.

Both the single period and multiperiod optimization policies include the self-financing condition as a constraint. However, the self-financing condition in the program is only $\sum_i u_t^i = 0$, but should include the costs as well: $\sum_i u_t^i + \phi(u_t)= 0$.

Continue getting error when trying to run the MultiPeriodOptimization example:

File "/home/xxx/.local/lib/python3.6/site-packages/cvxportfolio/simulator.py", line 116, in run_backtest
u = policy.get_trades(h, t)
File "/home/xxx/.local/lib/python3.6/site-packages/cvxportfolio/policies.py", line 351, in get_trades
z = cvx.Variable(*w.size)
TypeError: ABCMeta object argument after * must be an iterable, not numpy.int64

Does this example even work?

In current SinglePeriodOpt and MultiPeriodOpt, both returns the dollar traded for first time period.
This is OK for SinglePeriodOpt. But for MultiPeriodOpt, users are often interested in the complete trade schedule for every leg. As the result, it would be better if the get_trades() function returns a dictionary with enriched information, for example:
return {'portfolio_value': value,
'trade_schedule': z_vars.
'trade_times': trade_times}

Hi,

I went through the "Hello World" example via Jupyter notebook, but when defining SPO policy it does not recognize "return_forecast". After leaving this parameter out it says that the "alpha_model" needs to be given. I assumed that this is the same parameter as "return_forecast" and assigned "r_hat" to it, but this gives an AssertionError. Could you kindly tell me whether something has changed to the policy parameters?

NB - it seems the policies.py for the stand-alone module (v. 0.02 installed) differs from the policies.py on GitHub, which might be causing this issue.

Many thanks & best regards,

Mike Kraus

And Since the term leverage is overused and vague, I find the definition of gross exposure which is similar to leverage above except the weight of cash.
https://www.investopedia.com/terms/g/gross-exposure.asp

IMHO, the limit on gross exposure makes more sense than the leverage we used here. Consider a long-only and long-cash portfolio whose leverage is constant and gross exposure may vary. So I prefer to exclude weight of cash here.

Hi,

I've installed the packages pip. That's the only way I could get this package installed.
Can this package be updated with working examples please.

In the Singleperiod example,

simulated_tcost = cp.TcostModel(half_spread=0.0005/2., nonlin_coeff=1., sigma=sigmas, volume=volumes)

I get a ValueError as follows:
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

The current cvxportfolio uses Pandas.Panel data structure. According to Pandas development, Panel is deprecated and will be removed in future version. I do not know what this means. But for cvxportfolio, is there a plan to replace Panel data structure in future release?

I've tried running the nose tests but it appears the script tries to import pickle files when the data directory only has gzipped csv files. Perhaps the line in setup.py

package_data={'cvxportfolio': ['tests/sample_data.pickle']},

is the issue?

When running the HelloWorld example on macOS Catalina with a fresh download from the current repo with versions of the relevant libraries numpy=1.18.1, pandas=1.0.3, cvxpy=1.0.31, and python=3.7.7 on the cell

init_portfolio = pd.Series(index=returns.columns, data=250000.)
init_portfolio.USDOLLAR = 0
results = market_sim.run_multiple_backtest(init_portfolio,
                               start_time='2013-01-03',  end_time='2016-12-31',  
                               policies=[spo_policy, cp.Hold()])

I get the following error

<ipython-input-15-817c8501f3a4> in <module>
      4 results = market_sim.run_multiple_backtest(init_portfolio,
      5                                start_time='2013-01-03',  end_time='2016-12-31',
----> 6                                policies=[spo_policy, cp.Hold()])
      7 results[0].summary()

~/Desktop/cvxportfolio-master/cvxportfolio/simulator.py in run_multiple_backtest(self, initial_portf, start_time, end_time, policies, loglevel, parallel)
    150         if parallel:
    151             workers = multiprocess.Pool(num_workers)
--> 152             results = workers.map(_run_backtest, policies)
    153             workers.close()
    154             return results

~/anaconda3/envs/quantopt/lib/python3.7/site-packages/multiprocess/pool.py in map(self, func, iterable, chunksize)
    266         in a list that is returned.
    267         '''
--> 268         return self._map_async(func, iterable, mapstar, chunksize).get()
    269 
    270     def starmap(self, func, iterable, chunksize=None):

~/anaconda3/envs/quantopt/lib/python3.7/site-packages/multiprocess/pool.py in get(self, timeout)
    655             return self._value
    656         else:
--> 657             raise self._value
    658 
    659     def _set(self, i, obj):

Exception: Invalid dimensions for arguments.

Any insight into what may be causing this would be much appreciated. Thank you!

I have debugged the code and found out the root cause, cvx.multiply at cvxportfolio/policies.py: line number 257.
is causing this issue. How should I resolve this?

Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 1438, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/Users/ankitaggarwal/Applications/cvxportfolio/examples/HelloWorld.py", line 92, in
policies=[spo_policy, cp.Hold()])
File "/Users/ankitaggarwal/Applications/cvxportfolio/cvxportfolio/simulator.py", line 152, in run_multiple_backtest
results = workers.map(_run_backtest, policies)
File "/Users/ankitaggarwal/Applications/cvxportfolio/venv/lib/python3.7/site-packages/multiprocess/pool.py", line 268, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/Users/ankitaggarwal/Applications/cvxportfolio/venv/lib/python3.7/site-packages/multiprocess/pool.py", line 657, in get
raise self._value
ValueError: Cannot broadcast dimensions (562, 5) (5,)

Hello,

I've installed cvxportfolio following the instructions here: http://cvxportfolio.org/install/index.html.

When I run the nosetests, I get many errors saying the following:

INFO:root:Getting trades at time 2014-01-17 00:00:00
ERROR:root:The solver ECOS failed. Defaulting to no trades"

(with varying dates)

Despite the errors, the nosetests claim to have passed at the end:

---

Ran 18 tests in 33.872s

OK

The error sounds ominous though. Am I right to assume something has gone wrong?

Thanks in advance.

Hello, I installed the CVXPY and CVXPortfolio. When I run the Helloworld example, I got the following error message.

I am using windows 10, the CVXportfolio version is 0.0.12. Can someone know how to resolve this issue? Thanks!

RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\Users\jxl128031\AppData\Local\Continuum\anaconda3\lib\site-packages\multiprocess\pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\jxl128031\AppData\Local\Continuum\anaconda3\lib\site-packages\multiprocess\pool.py", line 44, in mapstar
    return list(map(*args))
  File "C:\Users\jxl128031\AppData\Local\Continuum\anaconda3\lib\site-packages\cvxportfolio\simulator.py", line 147, in _run_backtest
    policy, loglevel=loglevel)
  File "C:\Users\jxl128031\AppData\Local\Continuum\anaconda3\lib\site-packages\cvxportfolio\simulator.py", line 116, in run_backtest
    u = policy.get_trades(h, t)
  File "C:\Users\jxl128031\AppData\Local\Continuum\anaconda3\lib\site-packages\cvxportfolio\policies.py", line 266, in get_trades
    cost_expr, const_expr = cost.weight_expr(t, wplus, z, value)
  File "C:\Users\jxl128031\AppData\Local\Continuum\anaconda3\lib\site-packages\cvxportfolio\risks.py", line 58, in weight_expr
    self.expression = self._estimate(t, w_plus - self.w_bench, z, value)
  File "C:\Users\jxl128031\AppData\Local\Continuum\anaconda3\lib\site-packages\cvxportfolio\risks.py", line 105, in _estimate
    wplus, values_in_time(self.Sigma, t))
  File "C:\Users\jxl128031\AppData\Local\Continuum\anaconda3\lib\site-packages\cvxpy\atoms\quad_form.py", line 218, in quad_form
    raise Exception("Invalid dimensions for arguments.")
Exception: Invalid dimensions for arguments.
"""


The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "<ipython-input-2-0cb7372edff0>", line 6, in <module>
    policies=[spo_policy, cp.Hold()])

  File "C:\Users\jxl128031\AppData\Local\Continuum\anaconda3\lib\site-packages\cvxportfolio\simulator.py", line 152, in run_multiple_backtest
    results = workers.map(_run_backtest, policies)

  File "C:\Users\jxl128031\AppData\Local\Continuum\anaconda3\lib\site-packages\multiprocess\pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()

  File "C:\Users\jxl128031\AppData\Local\Continuum\anaconda3\lib\site-packages\multiprocess\pool.py", line 657, in get
    raise self._value

Exception: Invalid dimensions for arguments.

I am getting an exception when trying to use FactorMaxLimit and FactorMinLimit.

This is the minimal reproducible code:

import numpy as np
import pandas as pd
import cvxportfolio as cp

np.random.seed(100)

# spoof a factor loadings matrix with 100 stocks and 6 factors (e.g., think sectors)
factor_loadings = pd.get_dummies(pd.Series(np.random.randint(0,6,100)))

# create the expected returns DataFrame
r_hat_s = pd.Series(np.random.random(100))
r_hat_s.T['USDOLLAR']=0
r_hat = pd.DataFrame(r_hat_s, columns=[pd.Timestamp('2017-01-03')]).T

# create the prices DataFrame
prices_s = pd.Series(np.random.randint(20, 75, 100))
prices_noUSD = pd.DataFrame(prices_s, columns=[pd.Timestamp('2017-01-03')]).T
prices_s.T['USDOLLAR']=1.0
prices = pd.DataFrame(prices_s, columns=[pd.Timestamp('2017-01-03')]).T

# set up the optimization policy
spo_policy = cp.SinglePeriodOpt(
    return_forecast=r_hat,
    costs=[],
    constraints=[
        cp.LeverageLimit(1),
        cp.constraints.DollarNeutral(),
        cp.constraints.MaxWeights(0.10),
        cp.constraints.MinWeights(-0.10),
        cp.FactorMaxLimit(factor_loadings, 0.2),
        cp.FactorMinLimit(factor_loadings, -0.2)        
    ]
)

# setup empty starting portfolio
current_portfolio = pd.Series(index=r_hat.columns, data=0)
current_portfolio.USDOLLAR=100000

# run the single period optimization
shares_to_trade = spo_policy.get_rounded_trades(
    current_portfolio,
    prices,
    t=pd.Timestamp('2017-01-04')
)

If I eliminate the lines

        cp.FactorMaxLimit(factor_loadings, 0.2),
        cp.FactorMinLimit(factor_loadings, -0.2) 

then the code runs fine and produces the optimal trade list.

With the FactorMaxLimit, I get the following exception:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-542a0a7449e6> in <module>()
     36     current_portfolio,
     37     prices,
---> 38     t=pd.Timestamp('2017-01-04')
     39 )

/Users/jonathan/devwork/cvxportfolio/cvxportfolio/policies.pyc in get_rounded_trades(self, portfolio, prices, t)
     52         """Get trades vector as number of shares, rounded to integers."""
     53         return np.round(self.get_trades(portfolio,
---> 54                                         t) / time_locator(prices, t))[:-1]
     55 
     56 

/Users/jonathan/devwork/cvxportfolio/cvxportfolio/policies.pyc in get_trades(self, portfolio, t)
    269 
    270         constraints += [item for item in (con.weight_expr(t, wplus, z, value)
--> 271                                           for con in self.constraints)]
    272 
    273         for el in costs:

/Users/jonathan/devwork/cvxportfolio/cvxportfolio/policies.pyc in <genexpr>((con,))
    269 
    270         constraints += [item for item in (con.weight_expr(t, wplus, z, value)
--> 271                                           for con in self.constraints)]
    272 
    273         for el in costs:

/Users/jonathan/devwork/cvxportfolio/cvxportfolio/constraints.py in weight_expr(self, t, w_plus, z, v)
     46         if w_plus is None:
     47             return self._weight_expr(t, None, z, v)
---> 48         return self._weight_expr(t, w_plus - self.w_bench, z, v)
     49 
     50     @abstractmethod

/Users/jonathan/devwork/cvxportfolio/cvxportfolio/constraints.py in _weight_expr(self, t, w_plus, z, v)
    224         """
    225         #import pdb; pdb.set_trace()
--> 226         if isinstance(self.limit, pd.Series):
    227             limit = self.limit.loc[t]
    228         else:

/anaconda3/envs/py27/lib/python2.7/site-packages/pandas/core/ops.pyc in f(self, other, axis, level, fill_value)
   1265                 self = self.fillna(fill_value)
   1266 
-> 1267             return self._combine_const(other, na_op)
   1268 
   1269     f.__name__ = name

/anaconda3/envs/py27/lib/python2.7/site-packages/pandas/core/frame.pyc in _combine_const(self, other, func, errors, try_cast)
   3985         new_data = self._data.eval(func=func, other=other,
   3986                                    errors=errors,
-> 3987                                    try_cast=try_cast)
   3988         return self._constructor(new_data)
   3989 

/anaconda3/envs/py27/lib/python2.7/site-packages/pandas/core/internals.pyc in eval(self, **kwargs)
   3433 
   3434     def eval(self, **kwargs):
-> 3435         return self.apply('eval', **kwargs)
   3436 
   3437     def quantile(self, **kwargs):

/anaconda3/envs/py27/lib/python2.7/site-packages/pandas/core/internals.pyc in apply(self, f, axes, filter, do_integrity_check, consolidate, **kwargs)
   3327 
   3328             kwargs['mgr'] = self
-> 3329             applied = getattr(b, f)(**kwargs)
   3330             result_blocks = _extend_blocks(applied, result_blocks)
   3331 

/anaconda3/envs/py27/lib/python2.7/site-packages/pandas/core/internals.pyc in eval(self, func, other, errors, try_cast, mgr)
   1321             return block.eval(func, orig_other,
   1322                               errors=errors,
-> 1323                               try_cast=try_cast, mgr=mgr)
   1324 
   1325         # get the result, may need to transpose the other

/anaconda3/envs/py27/lib/python2.7/site-packages/pandas/core/internals.pyc in eval(self, func, other, errors, try_cast, mgr)
   1396 
   1397                 raise TypeError('Could not compare [%s] with block values' %
-> 1398                                 repr(other))
   1399 
   1400         # transpose if needed

TypeError: Could not compare [Expression(AFFINE, UNKNOWN, (100,))] with block values

This is failing on this line in constraints.py:

self.factor_exposure.T * w_plus[:-1] <= limit

I've tried to debug with no success. Any ideas on this?

Lastly, thank you for a great package!

@weiyialanchen

Hi,

I've encountered this error when trying to run the MultiPeriodOptimization.ipynb example notebook:

I'm using pandas 1.0.5 with pytables 3.6.1 in python 3.7.7.

import pandas as pd

datadir='~/git/cvxportfolio/data/'
risk_data = pd.HDFStore(datadir+'risk_model.h5', mode='r')
risk_data.get('factor_sigma')

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/usr/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py in _create_storer(self, group, format, value, encoding, errors)
   1587             try:
-> 1588                 cls = _STORER_MAP[pt]
   1589             except KeyError:

KeyError: 'wide'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-6-d64174533a2a> in <module>
----> 1 risk_data.get('factor_sigma')

/usr/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py in get(self, key)
    745         if group is None:
    746             raise KeyError(f"No object named {key} in the file")
--> 747         return self._read_group(group)
    748 
    749     def select(

/usr/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py in _read_group(self, group)
   1714 
   1715     def _read_group(self, group: "Node"):
-> 1716         s = self._create_storer(group)
   1717         s.infer_axes()
   1718         return s.read()

/usr/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py in _create_storer(self, group, format, value, encoding, errors)
   1588                 cls = _STORER_MAP[pt]
   1589             except KeyError:
-> 1590                 raise error("_STORER_MAP")
   1591             return cls(self, group, encoding=encoding, errors=errors)
   1592 

TypeError: cannot properly create the storer for: [_STORER_MAP] [group->/factor_sigma (Group) '',value-><class 'NoneType'>,format->None

I can successfully read idyos key in risk_model.h5 but not factor_sigma or exposures...

Any ideas folks? Am I missing something obvious here?

Thanks.

Not sure what is happening but I had to modify my values_in_time code to looks like the following to address the slicing with a null tau value (note the added "mi_slice" bit). Is anyone else noticing this?

    if hasattr(obj, '__call__'):
        return obj(t, tau)

    if isinstance(obj, pd.Series) or isinstance(obj, pd.DataFrame):
        try:
            if isinstance(obj.index, pd.MultiIndex):
                mi_slice = (t, tau)
                if (tau is None) or pd.isnull(tau):
                    mi_slice = (t, )
                return obj.loc[mi_slice]
            else:
                return obj.loc[t]
        except KeyError:
            return obj

    return obj