bdecon / us-chartbook Goto Github PK

Python script could generate tex code that creates an x tick mark with the latest available date.

Add chart showing AHE for total and production/non-supervisory.

Write some code that returns a dictionary of text for a given datetime.

When the latest quarter is available for some charts but not others, the chartbook should indicate that the charts are showing additional information.

Add text to describe chart showing NILF changes since March 2001.

Indicate on chartbook with mark/text when data shown in a chart was released in the last 7 days.

Add paragraph overview of investment from each sector (contribution to GDP from fixed investment).

For example bold if abs(gdp_contrib) > 2

Add one para to cover the major developments in retail sales data by category of store.

Show the age 25+ population by education, over time.

Look into better ways to retrieve data from the Fed. I'm fairly sure the current code will not work to retrieve updates.

To efficiently update the code, I want to run one command. This means I'll need to have a structured way to store the various individual chunks of code that update individual datasets or charts. Right now, there isn't really a structure to this.

GDP by industry will be released for 2019 Q2 in mid October. This section was produced relatively manually and needs to be fixed/sorted at some point.

Capital income: proprietor, interest and dividends, rent

Getting duplicate data from the sql database for a few series. I suspect the BEA tables that were not updated by the advance estimate get duplicated in the database.

Pretty sure the meaningful concept here will be private gross investment minus depreciation and not including inventories or inventory valuation adjustments. Need to research it a bit.

Optimize code in the contributions to GDP loop and add comments throughout. Convert some repetitive code to functions.

Add a new chart with the employment rate by state

Fundamental to the new chartbook is having meaningful text. I can't manually intervene in the process often, so I want a simple way to decide which text to show and to create the text correctly.

A basic option so far has been:

"In latest_date, series ((contributed/subtracted) latest_value (to/from)/did not contribute to) GDP growth, (compared to a (contribution/subtraction) of 3yr_ma_value percentage points/and did not contribute) on average over the past three years."

But that won't work for series other than contribution to growth and also is a bit confusing and probably not always helpful.

What I eventually want is to evaluate the latest value compared to its time series and use that evaluation to determine which text to generate. Specifically, I am interested in short- medium- and long-term developments and figuring out when they are meaningful.

Simple examples to target would be:

1. "((increased/decreased) by latest_value percentage points/was unchanged)"
2. "(latest_value percentage points (higher/lower)/unchanged)"
3. "((increased/decreased) to)/was unchanged at) latest_value"
   etc.

Some more advanced options:

1. Ideally, for some series, calculate the latest value and the last time it was that high/low. If that was more than X ago then use that line of text "(highest/lowest) since last_time_date"

2. Some cases have large recent changes: "an (increase/decrease) of change_value percentage points since reference_date"

3. Deciding when to use a different word, for example: near 0 change could be "virtually unchanged", or very large changes "a large (increase/decrease) of ..."

Both the text above the chart and the content of the chart currently involve manually interventions that can be automated.

Data collection structured by report/release.

Data processing--separate charts and text processing. Each chart must be of a identified chart type which has its own function. Charts each have their own notebook cell and name. Text handled based on chartbook section/topic/use cases to allow for more flexibility.

Convert the code that generates the age maps to a loop that produces both. Currently, each must be created manually.

Notebooks are not the answer for the final version of the chartbook. I need to move each data source's code into a self-contained .py file. Try this out with industrial production.

Current text doesn't make sense--it shows a fall in rented units but a positive contribution to total. Look into and fix.

The other category in US wealth is huge. It also is complicated because it handles the foreign sector balance. Try to make better sense of what is going on and include a paragraph to describe it.

Can either create a txt file with the info or try to get pgfplots to give me the data in the format I want. The latter is preferable but might not be feasible.

It's not clear to me what people mean when they say the US economy is shifting to services. On one hand, clearly production of goods has been outsourced relative to the past.

But also, there's a shift towards spending a much larger share of the national income on health and education. Yet the health indicators aren't improving and while increased education does increase wages, wages haven't been going up as quickly as in the past.

Create a template for adding bubble and scatter plots to the chartbook.

Add para on developments with USD exchange with EUR, GBP, JPY, and CAD.

Several charts use CPS data, so create a function that reads the files in the CPS data directly and identifies the most recent month.

I've been inconsistent in applying price adjustments, often using the PCE deflator when my general preference is for the CPI-U. At some point I will need to address this, and be more consistent in which deflator is used.

Use CPS ASEC to replicate the classic chart from Matt Bruenig.

Linking charts and tables to references would benefit from some structure/identification system beyond the name of the chart. Perhaps each chart can have a number like C1.1, C1.2, ..., C2.1, ... based on the section number. Tables would be T1.1...

The key would be automating the generation of each number and the link to the legend and references.

A calendar that stores dates of new releases/updates could drive the chartbook data updates.

Running the chartbook with only one half of the first section complete is already slow and in need of help from externalizing the graphics. Will need to figure out a good way to do this.

Figure out a better way to add the recession bars.

Ideally, I want to have the option to use a serif font in the chartbook. I tried to get this to work but didn't have any luck.

I'd like to know how expenses have been changing for US businesses. No idea on the source but ideally would want to know: labor, materials, capital, tax, and financing costs.

Current code would lose data / break if API request fails. Ideally I should store the data locally or keep a backup on GitHub. Look into options.

Be clear about API requests and results.

Add several charts to the personal income section. The obvious ones are for welfare and capital income by source. But more importantly try to make one good chart showing the distribution, possibly sourced from the CPS ASEC. Look at some past examples from Matt Bruenig, who has some of the clearest thinking on the topic.

The use of csv files for storing API returned data will not scale well and doesn't allow for storing metadata.

I'll need to figure out a structured way to store the API returned data and metadata. In past experience, a pickle file would probably work but SQL would give me a lot of powerful options. For example, I would eventually like to build in some features that tell me whether data has been updated recently and that also prevents a situation where good data is overwritten by empty or broken data.

A key consideration, though, needs to be ease of use later on. The current csv approach makes it very easy to get a variable into memory.

Full-time, part-time, and second job share of population by detailed occupation, from the CPS. (OCCD)

Add basic text to capture employment by occupation data.

Add summary paragraph for corporate profits

https://www.newyorkfed.org/microeconomics/hhdc/background.html

Use the table 7 data to replicate SPM figure 8, which shows how many people were removed from poverty by various programs.

Create a new template so that I can easily add horizontal bar charts where the values are stacked

Create a template for each chart type that can be filled by python.