fedspendingtransparency / usaspending-api Goto Github PK

After performing an advanced search for awards and opening the "Contract Summary" page for one award, then returning to the advanced search page via navigation bar and searching for a different award, the contract Summary Page showed the description field of the old award but all the other values of the new one.

New feature suggestion: the filter object should be able to accept agency, sub_agency and office codes. This would simplify queries, because the agency, sub_agency and offices names are capitalized in the CSV output file. Using these capitalized names in the filter object does not work. For example, filtering by sub_agency "MARITIME ADMINISTRATION" does not work, but "Maritime Administration" works.

Right now, I have to use the agency autocomplete end point to search for the proper capitalization of the agency name before using the filter object.

It should work the same way as with the NAICS and PSC codes for example.

I'm looking for a dictionary of potential values that the filter parameters may have.

For my purposes specifically, I'm hoping for a listing of the "award_type_codes" with human-readable names (i.e. not single letters). These codes in the endpoint documentation don't seem to match values in similarly names variables in the Data Dictionary - for instance, ContractAwardType.

For my own purposes, I'd ideally like to be able to filter to return only grants (rather than total obligations/spending). However, there's clearly a need for some sort of dictionary of these parameter values for others across the board.

I would like to only search by keyword on specific columns, especially only on award description. I see that this has been marked as a future enhancement. Do you have a timeline for implementation?

This is a new feature suggestion. The ability to retrieve the offices by sub-agency. Essentially, adding another sub-tier to the autocomplete for agencies and the /bulk_download/list_agencies/ end point.

When doing a GET to https://api.usaspending.gov/api/v2/financial_spending/object_class/?fiscal_year=2017&funding_agency_id=036, I keep getting an error message that says Missing one or more required query parameters: fiscal_year, funding_agency_id

I see this message in the browser and also when using Postman. Am I doing it wrong? 😀

Hello, the data dictionary link doesn't work. Does anyone have a copy? Thanks! Andrea

Perhaps this is somehow intended functionality but it makes the behavior of the custom awards download page quite confusing.

Ticking the 'Sub-Awards' box downloads from the subawards table, but this line I believe means that ticking the 'Prime Awards' box downloads from the transactions table, not the awards table as I expected, at least, as I was trying to get reference data for all awards and sub-awards rather than the detailed transactions data.

Instead I am having to issue a custom POST request and adding "awards" to the award_levels param rather than "prime_awards".

Hello, I'm looking at an example on https://api.usaspending.gov/docs/using-the-api that seems to suggest that double underscores should be usable with the "fields" option in a POST request:

{
    fields": ["toptier_agency__name", "subtier_agency__name"],
    "value": "DEFENSE",
    "mode": "contains",
    "limit": 100,
    "matched_objects": true
  "filters": []
}

Unfortunately this approach does not seem to work in my testing. Using this bit of json (examplequery.json)

{
    "limit": 1,
    "fields": ["recipient__location"],
    "filters": [
        {
            "field": "place_of_performance__state_code",
            "operation": "equals",
            "value": "CT"
        }
    ]
}

...and this curl invocation:

curl -H "Content-Type: application/json" -X POST --data-binary "@examplequery.json" https://api.usaspending.gov/api/v1/awards/ |jq  .|more

The API endpoint returns none of the recipient fields:

{
  "page_metadata": {
    "page": 1,
    "has_next_page": true,
    "has_previous_page": false,
    "next": "https://api.usaspending.gov/api/v1/awards/?limit=1&page=2&req=b0275bf10b3",
    "current": "https://api.usaspending.gov/api/v1/awards/?limit=1&page=1&req=b0275bf10b3",
    "previous": null
  },
  "req": "b0275bf10b3",
  "results": [
    {
      "type_description": "Definitive Contract",
      "description": "121NAG17S0010 INTL AID GRAIN"
    }
  ]
}

If, however, I do not attempt to specify any of the nested field names (i.e. "fields": ["recipient"], I can see that recipient__location is valid:

{
  "page_metadata": {
    "page": 1,
    "has_next_page": true,
    "has_previous_page": false,
    "next": "https://api.usaspending.gov/api/v1/awards/?limit=1&page=2&req=f2458c12573",
    "current": "https://api.usaspending.gov/api/v1/awards/?limit=1&page=1&req=f2458c12573",
    "previous": null
  },
  "req": "f2458c12573",
  "results": [
    {
      "type_description": "Definitive Contract",
      "description": "121NAG17S0010 INTL AID GRAIN",
      "recipient": {
        "legal_entity_id": 505323,
        "parent_recipient_unique_id": null,
        "recipient_name": "LOUIS DREYFUS COMPANY GRAINS MERCHANDISING LLC",
        "business_types": null,
        "business_types_description": "Unknown Types",
        "location": {
          "country_name": "UNITED STATES",
          "state_code": "CT",
          "state_name": null,
          "city_name": "WILTON",
          "address_line1": "40 DANBURY RD",
          "address_line2": null,
          "address_line3": null,
          "zip5": null,
          "foreign_postal_code": null,
          "foreign_province": null,
          "foreign_city_name": null,
          "location_country_code": "USA"
        }
      }
    }
  ]
}

Hi,

I was trying to do API call and limited time period to FY2017, but a lot of previous year's data were included (period_of performance_start_date WAY before 2016-10-01). Any reason it's doing that? This happens in the advanced search too. If I wanted to limit the grant that has starting date from 2016-10-01 to 2017-09-30, how should I do that (I followed the sample code, but also tried using period_of_performance_start_date instead)?

My next question is does the advanced search only include FY2008 and after data, because if I use the same filter, the results from advanced search and API call are different. API has results before FY2008 where advanced search doesn't have. For example, if you search "blandy experimental farm" as keywords, and advanced search has 6 results, but API call has 9 results. The extra three observations from API has period_of_performance_start_date of "2001-06-01", "2001-06-01", and "2005-03-01".

Last question is that API does not include the transaction data, correct? I would like to include variables like action_date but not a huge deal if it's unavailable.

Sorry about the paragraph, it will be great if you can answer the questions. Thanks a lot!

I'm trying to see where the information for POST functions data payloads are in the API docs. but I can't find them.

For example, I'm trying to find out /api/v2/search/spending_by_geography/ and what data payload I'm supposed to use for that API call, but the documentation is very sparse when it comes to those details? What i'm I supposed to pass in the API call in order to use the call? Where can I find this information? How can I use it?

Any help would be much appreciated.

I'm running into a constant issue with usaspending get and post requests. Often when there's at least one parameter passed into the request, the JSON returned will just be "message": "get() returned more than one RequestCatalog -- it returned x!" replacing x with any number from 2 to 4 or so. For example this happens when I add the page to the awards endpoint url. https://api.usaspending.gov/api/v1/accounts/awards/?page=1

I did a quick google and the internet suggests it's a django server issue, and can be fixed by replacing Model.objects.get() with Model.objects.filter() - https://stackoverflow.com/a/22064255

title

I have provided a generated file. If you download it, you can see multiple files with the same name packaged inside.

This would be fine with unique keys if the subaward data actually had something reliable you could combine into a composite key (since a combination of SubAward and PrimaryAward IDs can be duplicated). However, still not ideal, since I would not normally expect this behavior.

all_prime_awards_subawards_20190127061559.zip

When using "api/v2/download/awards", how do you access your zip file? I've tried putting in the generated url into my browser, but no zip file is ever generated or available for download.

Thanks!

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

I get empty results from this:

https://api.usaspending.gov/api/v2/financial_balances/agencies?funding_agency_id=95&fiscal_year=1998

Agency ID 95 is the Dept of Agriculture. They existed and spent money in FY1998, but unless I put 2017 for FY I get no results. What am I misunderstanding/doing wrong?

Hi, I have been hitting the Awards endpoint, https://api.usaspending.gov/api/v1/awards/ posting {"verbose": true, "limit":100} to retrieve all the data, and noticed that on page 2148, https://api.usaspending.gov/api/v1/awards/?limit=100&page=2148&req=c6e30f115d6 it returned a server 500 error. Is there any throttling that is being done that is being done that would return the error and is there anything that I can do to be able to retrieve all of the available data?

Thanks!

Alicia

Hello,

I was trying to use the Custom Award Bulk Download, but kept getting 500 error. Not sure what I have done wrong. Any idea?

The following link to the Recipient/Business Types returns "You do not have permission to update this wiki."

https://github.com/fedspendingtransparency/usaspending-api/wiki/Recipient-Business-Types

The incorrect link is located at:

https://github.com/fedspendingtransparency/usaspending-api/blob/master/usaspending_api/api_docs/api_documentation/search_filters.md#recipientbusiness-type

Does anyone already mirror or host the data or the API, what do we do when the website doesn't work for example, when the government is shutdown?

Curious what data APIs are available to get at the information found on a page such as https://www.usaspending.gov/#/award/9778360. We have a need in our app (for a USG agency) to search by award number and retrieve information such as the Recipient, Period of Performance, and contract types.

I've looked through https://api.usaspending.gov/docs/endpoints and the list doesn't seem to show what we're looking for (note that we don't want to download a ZIP file, but would rather be querying directly as the search on USASpending.gov seems to be doing.

Thanks a bunch!

Hello,

I've tried several examples of POST requests using the documentation and am only receiving 500 errors. Not sure if this is related to other issues.
Code:

import requests 

url_base = 'https://api.usaspending.gov'
endpoint = '/api/v2/download/awards/'

filter_object = {
    "filters": {"time_period": [
        {
            "start_date": "2001-01-01",
            "end_date": "2001-01-31"
        }
    ]},
    "columns": [
        "award_type",
        "awarding_agency_code"
    ]
}

request_url = url_base + endpoint
response = requests.post(request_url, data = filter_object)

response.reason
# Out[62]: 'Internal Server Error'

I've tried with other endpoints and filter objects, but POST requests always return with a 500.

I would like to be able to use certain parameters for an API call to get the exact information that I want from USA Spending. For example, I would like to get, from the awards endpoint, all the contracts for a state in a certain fiscal year( also if it is possible to get this for multiple years in a single API call ). Where can I find the parameter info and instructions on how to do this?

hi, I'm looking for API to find connection between parent recipient and kids recipient. I was not able to find it. need help

thanks!

Hi, thank you for making this api the idea is amazing!

I'm currently just messing around with the endpoints and get an internal server error whenever I try and access the /spending endpoint. Regardless of the type specified.

The only header that was set was for "Content-Type". I read the documentation again and I think I'm making the request correctly, if you think I am making an incorrect request please tell me where I'm going wrong.

Thank you!

When using the /search/spending_by_award/ endpoint, I am getting the following response with the example JSON request that is in the documentation:

{
    "detail": "Field 'filters|legal_entities' value '779928' is above max '5000'"
}

I was specifically wanting to query against legal_entities and had to remove award_ids to see the above error because if award_ids is in the filter set, it also causes an error:

{
    "detail": "Invalid value in 'filters|award_ids'. '1018950' is not a valid type (text)"
}

I'm trying to get the API working. I'm unable to get any requests to work. What is wrong with the command below?

$ curl --request GET --url https://www.usaspending.gov/api/v2/awards/66945037/

Response:

<html>
<head><title>404 Not Found</title></head>
<body>
<h1>404 Not Found</h1>
<ul>
<li>Code: NoSuchKey</li>
<li>Message: The specified key does not exist.</li>
<li>Key: #/404/index.html</li>
<li>RequestId: F1ABCD10DF4A934F</li>
<li>HostId: Mh1N4Ve/Gb3dLECkUhGBD7DAOWScL9Hp3FkAQDhgHTuxYogUdnbJaAxqWKetxgklWzdm6q/rUt8=</li>
</ul>
<hr/>
</body>
</html>

I supplied a JSON object like this:

    "award_levels": ["prime_awards"],
    "filters": {
        "keyword": "009256819"
    },
    "columns": ["award_id_piid", "parent_award_id", "federal_action_obligation", "total_dollars_obligated", "base_and_exercised_options_value", "current_total_value_of_award", "base_and_all_options_value", "potential_total_value_of_award", "action_date", "period_of_performance_start_date", "period_of_performance_current_end_date", "period_of_performance_potential_end_date", "awarding_agency_code", "awarding_agency_name", "awarding_sub_agency_code", "awarding_sub_agency_name", "awarding_office_code", "awarding_office_name", "funding_agency_code", "funding_agency_name", "funding_sub_agency_code", "funding_sub_agency_name", "funding_office_code", "funding_office_name", "foreign_funding", "foreign_funding_description", "recipient_duns", "recipient_name", "recipient_parent_name", "recipient_parent_duns", "recipient_country_code", "recipient_country_name", "recipient_address_line_1", "recipient_address_line_2", "recipient_city_name", "recipient_state_code", "recipient_state_name", "recipient_zip_4_code", "recipient_congressional_district", "recipient_phone_number", "recipient_fax_number", "primary_place_of_performance_country_code", "primary_place_of_performance_country_name", "primary_place_of_performance_city_name", "primary_place_of_performance_county_name", "primary_place_of_performance_state_code", "primary_place_of_performance_state_name", "primary_place_of_performance_zip_4", "award_description", "product_or_service_code_description", "dod_claimant_program_description", "naics_code", "naics_description"],
    "file_format": "csv"
}

I am only supplying 53 fields but was returned every field. Am I not specifying the columns correctly?

I am specifying the following call and it looks like the file generation will work:

{
    "award_levels": ["sub_awards"],
    "filters": {
        "keyword": "110483521"
    },
    "columns": [],
    "file_format": "csv"
}

However, once I go to check the file generation, I end up with this error:

I'm assuming this is a bug? Or is there something else causing this?

Thanks :)

I have provided a file that has a bad record.
Row 33 is entirely messed up and missing many fields at the beginning of the record, throwing everything off and breaking my processes.

all_contracts_prime_awards_1.zip

This is a test issue

I am getting a 403 unauthorized when trying to download zip files but not all the time.
Is there a limiter on how fast you can initiate downloads?

If so, I'd like to know the rules on that so I can put delays in my code.

Hi, I have been trying to get Contract Delivery Orders Calls by posting type = C to https://api.usaspending.gov/api/v1/awards/ and consistently encounter error on page 137 that suggests that my request has expired. I am requesting 100 records at a time and if I change that to 200 or 300, I immediately get back a 500 error.

body
{"verbose": true, "limit":100, "filters" : [{"field" : "type", "operation": "in", "value": ["C"]}]}

request url
https://api.usaspending.gov/api/v1/awards/?limit=100&page=137&req=c1b96461fb0

http error
HTTP/1.1 400 Bad Request - https://api.usaspending.gov/api/v1/awards/?limit=100&page=137&req=c1b96461fb0

message
{"message":"Requested 'req' 'c1b96461fb0' does not exist or has expired."}

Whenever I try to a post request with the API, I keep getting a 500 internal server error? Anyone else having the same issue?

What are the names of the fields in the API if you want to pull "number_of_offers_received" and "other_than_full_and_open_competition_code" from the v2/search/spending_by_award endpoint? I can't seem to find it in the documentation.

The api data dictionary (here) is missing definitions for many of the more technical (and obscure sounding) data elements, like "ussgl488100_upward_adjust_pri_undeliv_order_oblig_unpaid_cpe". It appears that many of these elements are defined in a file that (it looks like) @kaitlin and @bsweger put here: https://github.com/fedspendingtransparency/fedspendingtransparency.github.io/blob/master/_data/elements.csv . It would be useful to:

• Link to elements.csv from the API data dictionary page (or import the definitions from the _elements csv file to the in-page definitions)
• The elements.csv also explains the variable names used in the agency submission files (or, at least, I think it does). Assuming this is the correct file, maybe it could be linked from the top of the agency submission files pages, like this one ?

Thanks again for all y'alls work on this--this is really useful info!

I don't see the snapshot. Is it still available?

When trying to use /download/awards/, I get a response with a filename, but when I check /download/status/ I get this response:

{
    "detail": "Missing one or more required query parameters: award_levels"
}

The documentation does not address having to provide an award_level. I am trying to do a test request with a random DUNS:

{
    "filters": {
    	"recipient_search_text": ["113253443"]
    },
    "columns": [
        "award_type",
        "awarding_agency_code"
    ]
}

And what's worse, I get the same error message with the example JSON that is provided in the documentation.

Right now, restoring the snapshot requires spinning a 1.5 TB RDS instance which nominally cost $850/mo to maintain.

Would it be possible to make the snapshot available as a plain postgres dump file?

Hi,
I've tried several of the request examples for the API endpoints but a lot of the don't work.
For example,
/api/v2/bulk_download/list_monthly_files/ gives
400 Bad Request
/api/v2/search/spending_by_geography/ gives
500 Internal Server Error

I'm sending POST requests through Python. Is this an issue with the server or my end?

Thanks

More specifically, I am trying to filter by DUNS number and fiscal year. What are the filter names for this?

Is there any Swagger API documentation file built yet? Or is that something the API could serve us somehow? Fancy documentation and an API that knows the metadata well would assist with adopting the API. As would data mapping #341 (comment)

In looking through the data provided in the transactions endpoint, the place_of_performance is null for quite a lot of the transactions endpoint. Suggest fixing this for contracts by pulling place of performance from FPDS-NG using a combo of the PIID & modification number.

Example:
piid=SAQMMA16F0382&modification_number=M007

Per usaspending, place of performance is null except for country as USA.
Per fpds-ng, place of performance is Arlington, VA 22209

usaspending link: https://api.usaspending.gov/api/v1/transactions/?contract_data__piid=SAQMMA16F0382&modification_number=M007

fpds-ng link: https://www.fpds.gov/ezsearch/jsp/viewLinkController.jsp?agencyID=1900&PIID=SAQMMA16F0382&modNumber=M007&transactionNumber=0&idvAgencyID=1900&idvPIID=SAQMMA12D0168&actionSource=searchScreen&actionCode=&documentVersion=1.4&contractType=AWARD&docType=C

I am unable to locate the USAspending snapshot: arn:aws:rds:us-east-1:515495268755:snapshot:usaspending-db

Is the snapshot still being provided? Has the snapshot been renamed or moved? I am using a copy of the RDS database from one of the June shapshots and want to update the data with a snapshot from July but can no longer find the snapshot.

Please advise.

Thank you,
Bradley Wells

Hello, I am not very experienced with code and API setups, but I was trying to follow the readme on a windows system. I seem to have hit a roadblock when installing the packages defined by the requirements text file as it cannot install uWSGI.

I've tried to use Cygwin to get around that, but that is giving me trouble as well. Is there a way around using the uWSGI package? Has anyone successfully installed on a Windows environment that can give me some tips?

Thank you!

The query fails, when I specify columns to download with the:

/api/v2/download/awards

endpoint. It works fine when I set:

"columns": []

Full JSON options:

{"columns":["obligated_amount","product_or_service_code","product_or_service_code_description","naics_code","naics_description"],"filters":{"keywords":["transportation"," aviation"],"time_period":[{"start_date":"2018-06-30","end_date":"2019-06-30"}]}}

I do not get any error message. Just 500 Internal Server Error.

The fields that are returned from bulk_download/awards vs download/awards is a little different when it comes to the monetary fields:

A full list of fields in bulk_download but not in download is:
(I marked in bold the ones i think are most important, italics is secondary and normal is stuff that looks like fields that have to do with row status)

Contracts

modification_number
transaction_number
parent_award_modification_number
total_dollars_obligated
base_and_exercised_options_value
base_and_all_options_value
action_date
action_type_code
action_type
price_evaluation_adjustment_preference_percent_difference
undefinitized_action_code
undefinitized_action

These two fields reference obligations but I am assuming they are technically different values?
federal_action_obligation | obligated_amount

Assistance

(The missing fileds here seem a little more inconsequential, but I figured I would list them out anyway)
modification_number
action_date
correction_delete_indicator_code
correction_delete_indicator_description
action_type_code
action_type_description

These two fields reference obligations but I am assuming they are technically different values?
federal_action_obligation | obligated_amount

Is there anyway to get those other fields in the download/award endpoint like they are in the bulk_download/award?
Is this a bug or intended behavior?

Hello,

I tried filtering the data by specifying the date_type to "last_modified_date". However, this is not working as it still returns the data based on "action_date".

The endpoint I tried is api/v2/bulk_download/awards/. It also looks like for api/v2/download/awards/ and api/v2/download/transactions/, there is no option to specify the date_type and "action_date" is forced. I also tried downloading the data directly from the Download Center UI, and it has the same issue. Even if I choose last_modified_date, it still returns data by action_date.

Could someone look into this issue? I think it's important to have this feature to filter by Last Modified Date, so that the users can pick up any new updates to previous records.

Thanks very much!