Comments (41)
jpycroft
commented on June 3, 2024
1
Hi @nikhilwoodruff,
Yes, going for the full FRS might be the best solution.
I'll look into how to download. It seems that I'll need to agree to an End User Licence first ...
from og-uk.
jdebacker
commented on June 3, 2024
1
Also, when adding packages dependencies (like taxcalc or ogusa (neither of which should be necessary for OG-UK) or openfisca_uk), it would be best to install those as packages (via pip or conda-forge). Second best (or if you need to modify those packages) is to build the package on your machine via the source code (just be sure it's at the state you want it to be at).
The option you should avoid at all costs is to copy over all the source files and place them in a directory inside another project and call them as if they are just modules which is what is outlined above in the comment by @jpycroft that says:
Copy two files into the OG-UK folder:
a. ogusa
b. taxcalc (pip install taxcalc did not work. seemingly many errors:
This is dangerous because you will not be sure you have all the dependencies of the package, it will be difficult to version control, and you are not guaranteed things run.
FYI, current versions of taxcalc are no longer released on pip and need to be installed from conda-forge (conda install -c conda-forge taxcalc).
from og-uk.
jpycroft
commented on June 3, 2024
1
Hi @jdebacker, @nikhilwoodruff,
Let me first say about the sync-ing. Doing a upstream/main sync was the first thing I did. I assumed that was why I get different results than before. I was doing this:
(base) C:\Users\jonat\repos\OG-UK>git fetch upstream
(base) C:\Users\jonat\repos\OG-UK>git merge upstream/main
Already up to date.
OK, maybe this is the problem:
(base) C:\Users\jonat\repos\OG-UK>git remote show upstream
* remote upstream
Fetch URL: https://github.com/PSLmodels/OG-UK-Calibration.git
Push URL: https://github.com/PSLmodels/OG-UK-Calibration.git
HEAD branch: main
Remote branches:
gh-pages tracked
main tracked
Local ref configured for 'git push':
main pushes to main (fast-forwardable)
My upstream was still linked to the OG-UK-Calibration. Was that it? (If so, I still find that confusing for other reasons.) In any case, I'll do a delete and reinstall as you suggest Nikhil.
from og-uk.
nikhilwoodruff
commented on June 3, 2024
1
Thanks @jpycroft - good to know the synth FRS runs. That gcloud error looks strange - I should be able to have a proper look today before the OG meeting, but looks like it's been found by others too.
from og-uk.
jpycroft
commented on June 3, 2024
1
Hi @nikhilwoodruff,
Thanks again. The command (oguk-dev) C:\Users\jonat\repos>pip install --upgrade openfisca-uk gave a long list of "Requirement already satisfied" and ended with this:
Installing collected packages: argparse, openfisca-uk
Attempting uninstall: openfisca-uk
Found existing installation: OpenFisca-UK 0.20.3
Uninstalling OpenFisca-UK-0.20.3:
Successfully uninstalled OpenFisca-UK-0.20.3
Successfully installed argparse-1.4.0 openfisca-uk-0.20.4
So yes, it does give the "uninstalling openfisca-uk... installed openfisca-uk" message.
from og-uk.
jdebacker
commented on June 3, 2024
The matplotlib lines are warnings, I believe. It maybe that the OG-Core style file needs updating, or it could be an issue with trying to import this file from a URL as is done in both OG-USA and OG-UK (see here).
The real issue, and what is breaking execution, is
Number of workers = 4
Using baseline tax parameters from C:\Users\jonat\repos\OG-UK\examples\OG-UK-Example\OUTPUT_BASELINE\TxFuncEst_baseline.pkl
distributed.worker - WARNING - Compute Failed
Function: get_calculator_output
args: (True, 2018, None, 'cps')
kwargs: {}
Exception: KeyError("Unable to open object (object 'P_person_id' doesn't exist)")
Traceback (most recent call last):
File "run_oguk.py", line 176, in <module>
main()
File "run_oguk.py", line 60, in main
c = Calibration(p, estimate_tax_functions=True, client=client)
File "c:\users\jonat\repos\og-uk\oguk\calibrate.py", line 47, in __init__
tax_func_path=tax_func_path,
File "c:\users\jonat\repos\og-uk\oguk\calibrate.py", line 126, in get_tax_function_parameters
num_workers=num_workers,
File "c:\users\jonat\repos\og-uk\oguk\get_micro_data.py", line 138, in get_data
results = client.gather(futures)
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 1992, in gather
asynchronous=asynchronous,
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 833, in sync
self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\utils.py", line 340, in sync
raise exc.with_traceback(tb)
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\utils.py", line 324, in f
result[0] = yield future
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\tornado\gen.py", line 762, in run
value = future.result()
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 1851, in _gather
raise exception.with_traceback(traceback)
File "c:\users\jonat\repos\og-uk\oguk\get_micro_data.py", line 44, in get_calculator_output
sim = Microsimulation(year=year)
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\tools\simulation.py", line 243, in __init__
self.load_dataset(dataset, self.year)
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\tools\simulation.py", line 357, in load_dataset
"person": np.array(data["P_person_id"][year]),
File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\h5py\_hl\group.py", line 305, in __getitem__
oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py\h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'P_person_id' doesn't exist)"
distributed.worker - WARNING - Heartbeat to scheduler failed
(oguk-dev) C:\Users\jonat\repos\OG-UK\examples>
I've seen this issue with object 'P_person_id' doesn't exist before, but I can't recall exactly how it was resolved. It was some issue with the data used for OpenFisca-UK, I believe. Let me see if I can recreate.
from og-uk.
jdebacker
commented on June 3, 2024
@jpycroft I was able to generate this same error. I had thought I ran this example before, but not sure.
PR #27 maybe helpful here (though it needs to be updated with the master).
Also, it would be good to learn more about OpenFisca-UK. I'm still not totally sure how to do many things with the model and don't fully understand how the data is loaded and hierarchical data structures handled. It might be good for all of us working with OG-UK to learn more about the microsimulation model.
from og-uk.
jpycroft
commented on June 3, 2024
@jdebacker @nikhilwoodruff @rickecon
Continuing to try to understand why I getting the litany of errors with run_oguk.py.
I realise now that I'm not even getting past the import commands at the beginning of file. If I run to here:
import multiprocessing
from distributed import Client
import json
import time
import os,sys
from openfisca_core.model_api import Reform
from oguk.calibrate import Calibration
print('stop after oguk.calibrate')
sys.exit()
I already get a load of errors: see attached "run_oguk_stop_after_importCalibrate_11oct2021.txt"
Going to the next import command, and commenting out the previous import command, i.e.:
import multiprocessing
from distributed import Client
import json
import time
import os,sys
from openfisca_core.model_api import Reform
# from oguk.calibrate import Calibration
from ogcore import output_plots as op
print('stop after ogcore - output plots; Calibration off')
sys.exit()
I again get a load of errors: see attached "run_oguk_stop_after_ogcore-outputPlots_CalibrationOff_11oct2021.txt"
The same happens for the other "from ogcore" imports.
run_oguk_stop_after_importCalibrate_11oct2021.txt
run_oguk_stop_after_ogcore-outputPlots_CalibrationOff_11oct2021.txt
from og-uk.
jdebacker
commented on June 3, 2024
@jpycroft All those were warnings. Your set of commands did execute (see the stop after ogcore - output plots; Calibration off printed at the end).
But those messages are annoying. I just merged a PR to OG-Core and a PR to OG-UK that should fix this (they completely remove any imports of matplotlib style files from scripts used in the two packages). Install new versions of each and try that above again. That warnings should disappear.
from og-uk.
jpycroft
commented on June 3, 2024
Thanks @jdebacker!
I updated my branch. It hard to tell if there are fewer warnings. There are certainly still many (attached).
In any case, if it's only warnings, then perhaps the only important part of the output is the last lines:
usage: run_oguk.py [-h] reform
run_oguk.py: error: the following arguments are required: reform
I'll have a look at this later on.
run_oguk_after-poss-removing-some-warnings_12oct2021.txt
.
from og-uk.
jdebacker
commented on June 3, 2024
@jpycroft Did you reinstall the ogcore package? Did you rebuild the oguk package after updating your branch from the upstream main branch?
from og-uk.
jpycroft
commented on June 3, 2024
@jdebacker Thanks again!
I hadn't updated my OG-Core. I updated that and tried to install its environment. This only partially worked, ending with the following error message (full output attached):
Pip subprocess error:
ERROR: Could not install packages due to an OSError: [WinError 5] Acceso denegado: 'C:\\Users\\jonat\\AppData\\Local\\Temp\\pip-uninstall-lahmt6kq\\sphinx-apidoc.exe'
Consider using the `--user` option or check the permissions.
failed
CondaEnvException: Pip failed
In any case, I updated OG-UK again and updated the environment (oguk-dev). Trying to run "run_oguk.py" still results in the litany of warnings as before, culminating in the same error as above (full output attached):
usage: run_oguk.py [-h] reform
run_oguk.py: error: the following arguments are required: reform
og-core_try_to_build_environment_13oct2021.txt
run_oguk_after-updating_13oct2021.txt
from og-uk.
nikhilwoodruff
commented on June 3, 2024
Great to see it (at least partially) working @jpycroft - this one I can answer: I while ago I wrote this format for run_oguk.py to take as argument a reform object in a Python file - it should work with python run_oguk.py small_ubi_reform.ubi_reform (pointing to the ubi_reform object in small_ubi_reform.py). I can take it out if it's causing inconvenience.
from og-uk.
jdebacker
commented on June 3, 2024
@jpycroft, Nikhil pointed out the issue with your attempted run of the example script.
I think the matplotllib warnings are still there because OG-Core was not updated because your env failed to updated.
I'd remove the environment with: conda remove --name oguk-dev --all
Then create it anew with conda env create
from og-uk.
jdebacker
commented on June 3, 2024
@rickecon because he may have seen the same sphinx install error that Pycroft is seeing recently. Suggests perhaps removing those dependencies that just help with documentation.
from og-uk.
jpycroft
commented on June 3, 2024
Thanks for the ideas @nikhilwoodruff @jdebacker.
Running the python run_oguk.py small_ubi_reform.ubi_reform (having reinstalled the environment) I get the same litany of errors, but now I'm ending on the person_id error:
KeyError: "Unable to open object (object 'person_id' doesn't exist)"
Output attached.
Best, Jon
run_oguk_small_ubi_reform_23oct2021.txt
from og-uk.
jpycroft
commented on June 3, 2024
@rickecon, @jdebacker, @nikhilwoodruff,
Exciting news: I’m getting different error messages :-) Seriously, that is exciting!
Having updated to the latest environment.yml and doing “conda update conda”, the litany of hundreds of error messages is gone. I’ve worked my way through a number of error messages and seem now to be stuck on accessing the synthetic data set for OpenFiscaUK.
I’ll follow this up below with two more posts on:
- Adjustments I’ve already made to get past errors &
- Notes on error messages regarding the default / synthetic data set
from og-uk.
nikhilwoodruff
commented on June 3, 2024
Amazing - admittedly it's been a while since I last used the synthetic dataset, happy to debug anything else you get. Or, I don't suppose you can get access directly to the FRS by any means?
from og-uk.
jpycroft
commented on June 3, 2024
With reference to the thread above, #31 (comment)
I'm trying to run: python run_og_uk.py
Adjustments I’ve already made to get past errors:
-
change line in run_og_uk.py:
from og_uk_calibrate.calibrate import Calibration
--> from oguk.calibrate import Calibration -
Copy two files into the OG-UK folder:
a. ogusa
b. taxcalc (pip install taxcalcdid not work. seemingly many errors: "Preparing metadata (setup.py) ... error" ... "Command errored out with exit status 1"; then tries again with earlier version (e.g. 0.6.2.) and same message appears; also tried pip install taxcalc==0.4 / 0.4.1 / etc), Instead, fork & git clone Tax-Calculator. Then cut & paste "taxcalc" folder into OG-UK -
Re-install a few packages, which were missing because I had reinstalled Anaconda:
pip install numba
pip install eurostat
pip install xlsxwriter -
Remove command to put OG-USA version in parameters to save for reference
As ogusa is no longer part of the environment installation, the line in parameters.py:
self.ogusa_version = pkg_resources.get_distribution("ogusa").version
Throws an error:
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\pkg_resources\__init__.py", line 777, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'ogusa' distribution was not found and is required by the application
As it appears to be non-essential, for now I simply edited it out.
- The next error is a failure to find the default / synthetic data set; see next comment.
@jdebacker @rickecon @nikhilwoodruff
from og-uk.
jpycroft
commented on June 3, 2024
With reference to the thread above, #31 (comment) & #31 (comment)
I'm trying to run: python run_og_uk.py
Notes on error messages regarding the default / synthetic data set:
-
Error when running OG-UK:
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = 'C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk_data\microdata\openfisca_uk\frs_was_imp_None.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Checking the file location, all folders with \microdata\ are empty. -
Retrying to run OpenFiscaUK in isolation.
This was a suggestion from Nikhil that had previously worked, see #21 (comment) & #21 (comment).
Trying to do the same now, gives the error:
In [2]: sim = Microsimulation() # using the synthetic dataset
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-2-eef7f274daa0> in <module>
----> 1 sim = Microsimulation() # using the synthetic dataset
~\anaconda3\envs\test\lib\site-packages\openfisca_tools\microsimulation.py in __init__(self, reform, dataset, year)
40 self.dataset = dataset
41 if year is None:
---> 42 self.year = self.default_year or max(self.dataset.years)
43 else:
44 self.year = year
ValueError: max() arg is an empty sequence
This is a result of their being no default data set.
Possible solution
@nikhilwoodruff has already suggested (#31 (comment)) that, rather than worry about the synthetic data set, it may be more efficient to get access directly to the Family Resources Survey, which is what is needed to run properly anyway. I'm going ahead with the application for access now.
from og-uk.
nikhilwoodruff
commented on June 3, 2024
@jpycroft the full FRS would be ideal - good luck - but I wonder if the synthetic dataset works if you do this?
openfisca-uk-setup --set-default synth_frsfrom og-uk.
jpycroft
commented on June 3, 2024
Hi @nikhilwoodruff,
Thanks for the idea. When I do the openfisca-uk-setup --set-default synth_frs command, I get a different error, and I get it one command earlier in response to from openfisca_uk import Microsimulation: basically it complains that ModuleNotFoundError: No module named 'family_resources_survey'
Here's how I entered it, and the output below.
Best, Jon
(test) C:\Users\jonat\repos\openfisca-uk>openfisca-uk-setup --set-default synth_frs
Just setting default dataset, skipping questions.
(test) C:\Users\jonat\repos\openfisca-uk>ipython
Python 3.7.11 (default, Jul 27 2021, 09:42:29) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.29.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from openfisca_uk import Microsimulation
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-1-5d103e1b8098> in <module>
----> 1 from openfisca_uk import Microsimulation
~\repos\openfisca-uk\openfisca_uk\__init__.py in <module>
4 from openfisca_core.taxbenefitsystems import TaxBenefitSystem
5 from openfisca_uk import entities
----> 6 from openfisca_uk.microdata.simulation import Microsimulation
7 from openfisca_uk.tools.simulation import IndividualSim, PopulationSim
8 from openfisca_uk.reforms.presets.modelling import (
~\repos\openfisca-uk\openfisca_uk\microdata\simulation.py in <module>
7 from openfisca_uk.entities import *
8 from openfisca_core.simulation_builder import SimulationBuilder
----> 9 from openfisca_uk.microdata.frs.dataset import FRSDataset
10 from openfisca_uk.microdata.spi.dataset import SPIDataset
11 from openfisca_uk.microdata.frs.config import from_FRS
~\repos\openfisca-uk\openfisca_uk\microdata\frs\dataset.py in <module>
1 import numpy as np
2 import pandas as pd
----> 3 from family_resources_survey import FRS
4 from pathlib import Path
5 import shutil
ModuleNotFoundError: No module named 'family_resources_survey'
from og-uk.
jdebacker
commented on June 3, 2024
@jpycroft writes:
Adjustments I’ve already made to get past errors:
and then lists a bunch of changes he's made to source files to get things to work on his branch. But all of these changes are not relevant for the current versions of the sources files in the main branch.
@jpycroft I would suggest that you sync the branch you are working on with the upstream/main.
from og-uk.
nikhilwoodruff
commented on June 3, 2024
Thanks @jpycroft and @jdebacker - yes that's right. The code there is several months out of date. I think you shouldn't even need the repo cloned, but if you want to keep it, I'd suggest deleting it and reinstalling it.
from og-uk.
jdebacker
commented on June 3, 2024
@jpycroft Ah yes - the repo name change. That could be it. You'll want to reconfigure your remote URLs. Do that with a:
git remote set-url upstream https://github.com/PSLmodels/OG-UK.git
and, since your fork has this name also:
git remote set-url origin https://github.com/jpycroft/OG-UK.git
from og-uk.
jpycroft
commented on June 3, 2024
Thanks also for highlighting the conda-forge solution. I'll remember that if needed in future. (I should have clarified perhaps, the copying across was only ever intended as a stop gap to try and get moving at all.) Perhaps with the right version, it won't be required here.
from og-uk.
nikhilwoodruff
commented on June 3, 2024
Sounds good - though I did think that GitHub redirects name changes, so that actually probably should've worked. But not really a problem anymore
from og-uk.
jpycroft
commented on June 3, 2024
I removed and re-cloned OG-UK. I also removed and reinstalled the environment again. Now everything seems to be properly up-to-date.
The error I'm hitting now is that I'm missing the 'reform' argument:
(oguk-dev) C:\Users\jonat\repos\OG-UK\examples>python run_oguk.py
Using synthetic dataset.
usage: run_oguk.py [-h] reform
run_oguk.py: error: the following arguments are required: reform
The reform ought to be taken from reform=get_default_reform(), which is a simple UBI reform. It's not clear to me why this isn't being picked up.
Also I'm not clear what line 228 does
reform = getattr(__import__(python_module), object_name)
Any ideas would be welcome. Thanks!
from og-uk.
nikhilwoodruff
commented on June 3, 2024
OK, so that line essentially takes the argument, e.g. small_ubi_reform.ubi_reform, separates it into the module path (small_ubi_reform) and object name ubi_reform, and imports ubi_reform from its declaration in small_ubi_reform.py.
I wonder if either of the following works:
main(reform) -> main()on line 228 - er, actually this is line 228 on the repo rather than the one you quoted, perhaps you might be out of date?- Running
python run_oguk.py small_ubi_reform.ubi_reformfromOG-UK\
from og-uk.
jpycroft
commented on June 3, 2024
Hi @nikhilwoodruff,
Thanks for the suggestions and clarification.
Switching from main(reform) -> main(), it still wants a reform:
(btw, I'd added a print above, which explains the line change.)
(oguk-dev) C:\Users\jonat\repos\OG-UK>python examples/run_oguk.py
Using synthetic dataset.
confirm: name does equal __main__
usage: run_oguk.py [-h] reform
run_oguk.py: error: the following arguments are required: reform
With main() or main(reform), I get this output running with small_ubi_reform.ubi_reform:
(oguk-dev) C:\Users\jonat\repos\OG-UK>python examples/run_oguk.py small_ubi_reform.ubi_reform
Using synthetic dataset.
confirm: name does equal __main__
Using synthetic dataset.
Using synthetic dataset.
Using synthetic dataset.
Using synthetic dataset.
Number of workers = 4
Using baseline tax parameters from C:\Users\jonat\repos\OG-UK\examples\OG-UK-Example\OUTPUT_BASELINE\TxFuncEst_baseline.pkl
distributed.worker - WARNING - Compute Failed
Function: get_calculator_output
args: (True, 2018, None, 'frs')
kwargs: {}
Exception: ValueError('max() arg is an empty sequence')
Traceback (most recent call last):
File "examples/run_oguk.py", line 229, in <module>
main(reform)
File "examples/run_oguk.py", line 96, in main
c = Calibration(p, estimate_tax_functions=True, client=client)
File "c:\users\jonat\repos\og-uk\oguk\calibrate.py", line 41, in __init__
tax_func_path=tax_func_path,
File "c:\users\jonat\repos\og-uk\oguk\calibrate.py", line 120, in get_tax_function_parameters
num_workers=num_workers,
File "c:\users\jonat\repos\og-uk\oguk\get_micro_data.py", line 191, in get_data
results = client.gather(futures)
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 1992, in gather
asynchronous=asynchronous,
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 833, in sync
self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\utils.py", line 340, in sync
raise exc.with_traceback(tb)
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\utils.py", line 324, in f
result[0] = yield future
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\tornado\gen.py", line 762, in run
value = future.result()
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 1851, in _gather
raise exception.with_traceback(traceback)
File "c:\users\jonat\repos\og-uk\oguk\get_micro_data.py", line 94, in get_calculator_output
sim = Microsimulation()
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_tools\microsimulation.py", line 42, in __init__
self.year = self.default_year or max(self.dataset.years)
ValueError: max() arg is an empty sequence
So, it appears to be recognising the reform. It states that it's going to use the synthetic data set, but fails to find the default data.
from og-uk.
jpycroft
commented on June 3, 2024
Hi @nikhilwoodruff,
Back to trying to get the model running with FRS. Finally got round to trying this out over the weekend.
First, OG-UK does begin to run with synthetic data. To run with FRS, you mentioned to
i. pip install openfisca-uk - worked fine for me - then
ii. run openfisca-uk
I'm not clear which command to use to get openfisca-uk to run. I first tried the command suggested by OG-UK:
openfisca-uk-data frs_enhanced download 2019
which gave the error below:
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
WARNING:google.auth._default:Authentication failed using Compute Engine authentication due to unavailable metadata server.
INFO:root:Could not automatically authenticate with Google Cloud, prompting login...
Do I need to set up the FRS data somewhere, or be logged in to UKdata?
I tried various permutations to the command to run openfisca, but haven't hit on a combination that works.
If you have any thoughts, please share, and hope to catch you later at the meeting.
from og-uk.
nikhilwoodruff
commented on June 3, 2024
As per that issue could you try running gcloud auth application-default login?
from og-uk.
jpycroft
commented on June 3, 2024
Hi @nikhilwoodruff,
Thanks for the idea! I get this:
(oguk-dev) C:\Users\jonat\repos\OG-UK>gcloud auth application-default login
"gcloud" no se reconoce como un comando interno o externo,
programa o archivo por lotes ejecutable.
Translation: "gcloud is not recognised as an internal or external command, executable programme or batch file."
I pip installed gcloud and then google-cloud (though the pypi docs say that neither is currently maintained).
In any case, the error message stays the same.
from og-uk.
jpycroft
commented on June 3, 2024
Hi @nikhilwoodruff,
Thanks for adding this change: https://github.com/PolicyEngine/openfisca-tools/blob/master/setup.py
I did a "pip install openfisca-uk" should update my version to try to incorporate it. Unfortunately, I'm still getting the same output. Is there something else I need to run first?
(oguk-dev) C:\Users\jonat\repos\OG-UK>gcloud auth application-default login
"gcloud" no se reconoce como un comando interno o externo,
programa o archivo por lotes ejecutable.
(oguk-dev) C:\Users\jonat\repos\OG-UK>openfisca-uk-data frs_enhanced download 2019
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
WARNING:google.auth._default:Authentication failed using Compute Engine authentication due to unavailable metadata server.
INFO:root:Could not automatically authenticate with Google Cloud, prompting login...
from og-uk.
nikhilwoodruff
commented on June 3, 2024
Ah thanks @jpycroft - sorry, could you try pip install --upgrade openfisca-uk now? I've added google-cloud-storage as a dependency, so it should get installed (might be worth running pip install google-cloud-storage after, just to be sure, but this should happen automatically).
One of the differences between my local setup and yours is that I have the gcloud CLI installed. But I'm aiming for that to not be required to download the microdata.
from og-uk.
jpycroft
commented on June 3, 2024
Hi @nikhilwoodruff,
I get a different output now, which is surely progress!
I entered:
- pip install --upgrade openfisca-uk
- pip install google-cloud-storage (though everything was "Requirement already satisfied" as you thought it would be.)
Then openfisca-uk-data frs_enhanced download 2019 gives this:
(oguk-dev) C:\Users\jonat\repos\OG-UK>openfisca-uk-data frs_enhanced download 2019
Traceback (most recent call last):
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\jonat\anaconda3\envs\oguk-dev\Scripts\openfisca-uk-data.exe\__main__.py", line 4, in <module>
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\__init__.py", line 6, in <module>
from openfisca_uk.tools.simulation import (
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\tools\simulation.py", line 3, in <module>
from openfisca_uk.data import EnhancedFRS
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\__init__.py", line 1, in <module>
from openfisca_uk.data.datasets import (
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\datasets\__init__.py", line 1, in <module>
from openfisca_uk.data.datasets.frs import (
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\datasets\frs\__init__.py", line 1, in <module>
from .frs import FRS
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\datasets\frs\frs.py", line 2, in <module>
from openfisca_tools.data import PrivateDataset
ModuleNotFoundError: No module named 'openfisca_tools.data'
from og-uk.
nikhilwoodruff
commented on June 3, 2024
Great stuff @jpycroft - the good news is that I know exactly what this is, and I think it should be fixed now if you update OpenFisca-UK (for context, this issue was caused by me naively removing MANIFEST.in from the repo without realising what it did: PolicyEngine/policyengine-uk#574).
from og-uk.
jpycroft
commented on June 3, 2024
Thanks again @nikhilwoodruff!
I redid the pip install --upgrade openfisca-uk. (I think) everything was "Requirement already satisfied" except for:
Installing collected packages: argparse
Successfully installed argparse-1.4.0
Then the same command as before (I think) still gives the same output. Any other ideas?
(oguk-dev) C:\Users\jonat\repos\OG-UK>openfisca-uk-data frs_enhanced download 2019
Traceback (most recent call last):
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\jonat\anaconda3\envs\oguk-dev\Scripts\openfisca-uk-data.exe\__main__.py", line 4, in <module>
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\__init__.py", line 6, in <module>
from openfisca_uk.tools.simulation import (
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\tools\simulation.py", line 3, in <module>
from openfisca_uk.data import EnhancedFRS
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\__init__.py", line 1, in <module>
from openfisca_uk.data.datasets import (
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\datasets\__init__.py", line 1, in <module>
from openfisca_uk.data.datasets.frs import (
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\datasets\frs\__init__.py", line 1, in <module>
from .frs import FRS
File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\datasets\frs\frs.py", line 2, in <module>
from openfisca_tools.data import PrivateDataset
ModuleNotFoundError: No module named 'openfisca_tools.data'
from og-uk.
nikhilwoodruff
commented on June 3, 2024
Sorry for my delay in getting back to this @jpycroft! Maybe we can debug it a bit at the meeting tomorrow? One thought: running the --upgrade flag should result in some message at the end saying "uninstalling openfisca-uk... installed openfisca-uk", did this appear?
from og-uk.
jpycroft
commented on June 3, 2024
Hi @nikhilwoodruff,
Actually, it's a bit disappointing ... it didn't quite solve the issue. Here's what happened.
First, pip installing openfisca-uk within the newly created "openfisca-uk" environment (with Python 3.7.13) did eventually work. Output attached as txt.
pip install openfisca-uk_25apr2022.txt
Then the command to download 2019 frs data gave the following:
(openfisca-uk) C:\Users\jonat\repos>openfisca-uk-data enhanced_frs download 2019
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
WARNING:google.auth._default:Authentication failed using Compute Engine authentication due to unavailable metadata server.
INFO:root:Could not automatically authenticate with Google Cloud, prompting login...
Encountered an error:
Traceback (most recent call last):
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_tools\data\private.py", line 19, in _get_storage_bucket
client = storage.Client()
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\google\cloud\storage\client.py", line 165, in __init__
_http=_http,
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\google\cloud\client\__init__.py", line 320, in __init__
_ClientProjectMixin.__init__(self, project=project, credentials=credentials)
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\google\cloud\client\__init__.py", line 268, in __init__
project = self._determine_default(project)
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\google\cloud\client\__init__.py", line 287, in _determine_default
return _determine_default_project(project)
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\google\cloud\_helpers\__init__.py", line 152, in _determine_default_project
_, project = google.auth.default()
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\google\auth\_default.py", line 579, in default
raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\Scripts\openfisca-uk-data.exe\__main__.py", line 7, in <module>
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_uk\data\cli.py", line 6, in cli
openfisca_data_cli(DATASETS)
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_tools\data\cli.py", line 33, in openfisca_data_cli
raise e
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_tools\data\cli.py", line 26, in openfisca_data_cli
result = target(*args.args)
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_tools\data\dataset.py", line 69, in wrapper
return fn(*args, **kwargs)
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_tools\data\private.py", line 55, in download
bucket = self._get_storage_bucket()
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_tools\data\private.py", line 28, in _get_storage_bucket
stderr=subprocess.DEVNULL,
File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['gcloud auth application-default login']' returned non-zero exit status 1.
PS: I retried the gcloud auth, but same error as before:
(openfisca-uk) C:\Users\jonat\repos>gcloud auth application-default login
"gcloud" no se reconoce como un comando interno o externo,
programa o archivo por lotes ejecutable.
from og-uk.
nikhilwoodruff
commented on June 3, 2024
Ah, well thanks for getting through the installation: I think it's still a step in the right direction. The error makes sense to me: it's because the gcloud CLI is not installed. I've tried to avoid it by trying to find ways to not require its installation, but at this point it seems inevitable unfortunately. So, could you please install the tool on your machine and try it again? Thanks.
from og-uk.
Related Issues (20)
- Change directory and environment names HOT 1
- OG-UK set up HOT 27
- Unable to solve the steady-state equilibrium HOT 5
- Can't find the steady-state with DEP tax function HOT 17
- Conda environment fails to create HOT 9
- Allow for multiple budget years HOT 1
- Highly negative MTRs HOT 5
- Possible 25x speedup in steady-state `inner_loop` function
- [Security] Workflow check_black.yml is using vulnerable action actions/checkout
- GH actions issues
- Very long matplotlib warning on run of example script HOT 1
- Solving the steady-state HOT 4
- Documentation of OpenFisca-UK describes Tax-Calculator HOT 2
- Something is broken in download option in eurostat package HOT 2
- UK proposed corporate tax reform results
- Update OpenFisca-UK to PolicyEngine-UK HOT 1
- Update `calibrate.py` due to new tax function parameters
- Add UK synthetic dataset
- Updates to be compatible with latest OG-Core HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from og-uk.