Code Monkey home page Code Monkey logo

biosample-xmldb-sqldb's People

Contributors

turbomam avatar

Watchers

 avatar  avatar  avatar

Forkers

eecavanna

biosample-xmldb-sqldb's Issues

database competency questions

What claims would we like to make about the Biosample metadata presented in the NMDC MongoDB compared to NCBI?

Acknowledge that our Biosample Postgres database may be lossy esp wrt non-attribute metadata. There are some paths that aren't ingested at all and others that result in concatenations (which are harder to search over efficiently)

Break compound questions like "how many soil metagenome samples have a valid latitude and longitude" into prerequisites like

  • can we determine whether a Biosample is a soil sample?
  • can we determine whether a Biosample contains a "metagenome" compatible microbial community
  • do we need to link to other systems to make these decisions?

sparsity analysis of NCBI Biosamples via relational database created by this repo

@ssarrafan @emileyfadrosh and others from NMDC went through milestones and priorities and wanted to follow up something that's not a specific milestone but is a priority for Emiley's talk in June.

The request is for @turbomam to summarize the relational-structured NCBI Biosamples database with the goal of showing how much data is there and how sparse the metadata is and how much more valuable the data would be with metadata.

  • How much data is in there
    • just a row count of Biosamples, or broken out by packages/checklists/whatever?
  • sparsity
    • presumably we wouldn't want to consider "missing", empty strings etc useful values, and therefor replace them with NULL before calculating sparsity. There are lots of string that indicate missing values in there, so we should probably prioritize the most common ones.
  • what about metadata that isn't in a machine actionable format?

"/srv/basex/shared-chunks/biosample_set_from_37000001.xml" (Line 20346097): XML document structures must start and end within the same entity.

head shared-chunks/biosample_set_from_37000001.xml
<?xml version="1.0" encoding="UTF-8"?>
<BioSampleSet>
<BioSample access="public" publication_date="2023-12-27T00:00:00.000" last_update="2023-12-27T14:33:14.303" submission_date="2023-12-27T14:10:10.597" id="39145858" accession="SAMN39145858">
  <Ids>
    <Id db="BioSample" is_primary="1">SAMN39145858</Id>
    <Id db_label="Sample name">CO-CDPHE-41544812</Id>
    <Id db="SRA">SRS20005259</Id>
  </Ids>
  <Description>
    <Title>PCR tiled amplicon WGS of SARS-CoV-2</Title>
tail shared-chunks/biosample_set_from_37000001.xml
    <Attribute attribute_name="geo_loc_name" harmonized_name="geo_loc_name" display_name="geographic location">USA</Attribute>
    <Attribute attribute_name="lat_lon" harmonized_name="lat_lon" display_name="latitude and longitude">missing</Attribute>
    <Attribute attribute_name="host" harmonized_name="host" display_name="host">missing</Attribute>
    <Attribute attribute_name="host_disease" harmonized_name="host_disease" display_name="host disease">missing</Attribute>
  </Attributes>
  <Links>
    <Link type="entrez" target="bioproject" label="PRJNA230403">230403</Link>
  </Links>
  <Status status="live" when="2024-02-12T02:56:05.936"/>
</BioSample>

More FTS indices on tables other than NCBI Biosample data

Tables with data about Biosamples from GOLD

  • gold_sequencing_project

semantic-sql tables about ontology terms, esp. EnvO

  • entailed_edge
  • prefix
  • statements

INSDC controlled vocabulary

  • insdc_country_values
  • insdc_missing_data_list
  • insdc_missing_data_table

MIxS model

  • mixs_class_definitions
  • mixs_global_slots

NCBI model

  • ncbi_package_attribute_use
  • ncbi_package_definitions
  • ncbi_metagenomes

NCBI to MIxS model alignment

  • ncbi_mixs_class_mappings

Tables with data about Biosamples from NCBI

  • ncbi_attributes_all_long
  • ncbi_attributes_harmonized_wide
  • non_attribute_metadata

crashed during pivot_harmonized_attributes.py. Out of RAM (on host?)

poetry run python biosample_xmldb_sqldb/pivot_harmonized_attributes.py
accessible_tables = ['all_ncbi_attributes_long', 'non_attribute_metadata']
Processing through row 0 of 549805612 at 2024-02-13T02:52:40.573700
pivoted_data.shape = (71155, 159)
...
Processing through row 33000000 of 549805612 at 2024-02-13T03:25:32.826314
pivoted_data.shape = (2280496, 531)
Processing through row 34000000 of 549805612 at 2024-02-13T03:27:08.118753
make: *** [Makefile:170: postgres-pivot] Killed

NCBI Package/Attribute use

import xml.etree.ElementTree as ET
import sqlalchemy
from sqlalchemy import create_engine, MetaData
import csv
import pandas as pd

INPUT_FILE = 'ncbi_biosample_package_definitions.xml'
OUTPUT_FILE = 'ncbi_biosample_package_definitions.tsv'   

headers = ['Name', 'DisplayName','EnvPackage', 'EnvPackageDisplay', 'Example', 'NotAppropriateFor', 'ShortName', 'group', 'Description', 'attributes']

# df = pd.DataFrame(columns=headers)

lod = []

destination_table_name = 'ncbi_package_definitions'

engine = create_engine(
    'postgresql://postgres:<SECRET>@localhost:15432/ncbi_biosamples_feb26')

# connection = engine.connect()


with open(INPUT_FILE) as f:
    xml = f.read()

root = ET.fromstring(xml)

with open(OUTPUT_FILE, 'w', newline='') as f:
    # dict_writer = csv.DictWriter(f, delimiter='\t', fieldnames=headers)
    # dict_writer.writeheader()

    for package in root.findall('Package'):
        row = {}
        for field in headers:
            if field == 'group':
                row[field] = package.get('group')
            
            elif field == 'attributes':
                attrs = []
                for key, value in package.items():  
                    if key != 'group':
                        attrs.append(f"{key}={value}")
                row[field] = "|".join(attrs)
            
            else:
                row[field] = package.findtext(field)

            lod.append(row)


lod_frame = pd.DataFrame(lod)

print(lod_frame)

lod_frame.to_sql(destination_table_name, engine, index=False, if_exists='replace')

record current DDL

-- DROP SCHEMA public;

CREATE SCHEMA public AUTHORIZATION pg_database_owner;
-- public.entailed_edge definition

-- Drop table

-- DROP TABLE public.entailed_edge;

CREATE TABLE public.entailed_edge (
	subject text NULL,
	predicate text NULL,
	"object" text NULL
);
CREATE INDEX idx_190378_entailed_edge_sp ON public.entailed_edge USING btree (subject, predicate);
CREATE INDEX idx_190378_entailed_edge_spo ON public.entailed_edge USING btree (subject, predicate, object);


-- public.gold_sequencing_project definition

-- Drop table

-- DROP TABLE public.gold_sequencing_project;

CREATE TABLE public.gold_sequencing_project (
	project_gold_id text NULL,
	project_name text NULL,
	sequencing_strategy text NULL,
	its_proposal_id float8 NULL,
	its_sequencing_project_id float8 NULL,
	its_sample_id float8 NULL,
	pmo_project_id float8 NULL,
	ncbi_bioproject_accession text NULL,
	ncbi_biosample_accession text NULL,
	sra_experiment_ids text NULL,
	project_genome_publication_pubmed_id text NULL,
	project_other_publication_pubmed_id text NULL,
	project_status text NULL,
	sequencing_status text NULL,
	sequencing_centers text NULL,
	project_funding text NULL,
	project_legacy_gold_id text NULL,
	study_gold_id text NULL,
	organism_gold_id text NULL,
	biosample_gold_id text NULL,
	project_contacts text NULL,
	project_jgi_data_utilization_status text NULL,
	project_jgi_award_dois text NULL
);
CREATE INDEX gold_sequencing_project_ncbi_biosample_accession_idx ON public.gold_sequencing_project USING btree (ncbi_biosample_accession, biosample_gold_id, project_gold_id, ncbi_bioproject_accession);


-- public.insdc_country_values definition

-- Drop table

-- DROP TABLE public.insdc_country_values;

CREATE TABLE public.insdc_country_values (
	insdc_country text NULL
);
CREATE INDEX insdc_country_values_insdc_country_idx ON public.insdc_country_values USING btree (insdc_country);


-- public.insdc_missing_data_list definition

-- Drop table

-- DROP TABLE public.insdc_missing_data_list;

CREATE TABLE public.insdc_missing_data_list (
	missing_value_term varchar NULL
);
CREATE INDEX insdc_missing_data_list_missing_value_term_idx ON public.insdc_missing_data_list USING btree (missing_value_term);


-- public.insdc_missing_data_table definition

-- Drop table

-- DROP TABLE public.insdc_missing_data_table;

CREATE TABLE public.insdc_missing_data_table (
	top_level text NULL,
	lower_level text NULL,
	lower_level_definition text NULL,
	reporting_level text NULL,
	reporting_definition text NULL
);
CREATE INDEX insdc_missing_data_table_top_level_idx ON public.insdc_missing_data_table USING btree (top_level, lower_level, reporting_level);


-- public.mixs_class_definitions definition

-- Drop table

-- DROP TABLE public.mixs_class_definitions;

CREATE TABLE public.mixs_class_definitions (
	"class" text NULL,
	aliases text NULL,
	class_uri text NULL,
	"comments" text NULL,
	description text NULL,
	from_schema text NULL,
	in_subset text NULL,
	is_a text NULL,
	mixin text NULL,
	mixins text NULL,
	title text NULL,
	tree_root text NULL
);
CREATE INDEX mixs_class_definitions_class_idx ON public.mixs_class_definitions USING btree (class, class_uri, is_a, mixins);


-- public.mixs_global_slots definition

-- Drop table

-- DROP TABLE public.mixs_global_slots;

CREATE TABLE public.mixs_global_slots (
	slot text NULL,
	"examples values" text NULL,
	structured_pattern text NULL,
	"Expected_value" text NULL,
	"Preferred_unit" text NULL,
	"comments" text NULL,
	description text NULL,
	from_schema text NULL,
	in_subset text NULL,
	keywords text NULL,
	multivalued text NULL,
	pattern text NULL,
	"range" text NULL,
	recommended text NULL,
	required text NULL,
	slot_uri text NULL,
	string_serialization text NULL,
	title text NULL
);
CREATE INDEX mixs_global_slots_slot_idx ON public.mixs_global_slots USING btree (slot, in_subset, range, recommended, required, slot_uri, title);


-- public.ncbi_attributes_all_long definition

-- Drop table

-- DROP TABLE public.ncbi_attributes_all_long;

CREATE TABLE public.ncbi_attributes_all_long (
	raw_id int8 NULL,
	attribute_name text NULL,
	harmonized_name text NULL,
	display_name text NULL,
	unit text NULL,
	value text NULL
);
CREATE INDEX names_fts ON public.ncbi_attributes_all_long USING gin (to_tsvector('english'::regconfig, ((((attribute_name || ' '::text) || harmonized_name) || ' '::text) || display_name)));
CREATE INDEX ncbi_attributes_all_long_harmonized_name_idx ON public.ncbi_attributes_all_long USING btree (harmonized_name);
CREATE INDEX ncbi_attributes_all_long_raw_id_idx ON public.ncbi_attributes_all_long USING btree (raw_id);
CREATE INDEX value_fts ON public.ncbi_attributes_all_long USING gin (to_tsvector('english'::regconfig, value));


-- public.ncbi_attributes_harmonized_wide definition

-- Drop table

-- DROP TABLE public.ncbi_attributes_harmonized_wide;

CREATE TABLE public.ncbi_attributes_harmonized_wide (
	raw_id int4 NOT NULL,
	abs_air_humidity text NULL,
	additional_info text NULL,
	add_recov_method text NULL,
	address text NULL,
	adjacent_environment text NULL,
	adj_room text NULL,
	affection_status text NULL,
	age text NULL,
	agrochem_addition text NULL,
	air_pm_concen text NULL,
	air_temp text NULL,
	air_temp_regm text NULL,
	alkalinity text NULL,
	alkalinity_method text NULL,
	alkyl_diethers text NULL,
	al_sat text NULL,
	al_sat_meth text NULL,
	altitude text NULL,
	aminopept_act text NULL,
	ammonium text NULL,
	amniotic_fluid_color text NULL,
	analyte_type text NULL,
	anamorph text NULL,
	ances_data text NULL,
	animal_body_cond text NULL,
	animal_diet text NULL,
	animal_env text NULL,
	animal_intrusion text NULL,
	animal_sex text NULL,
	anim_water_method text NULL,
	annual_precpt text NULL,
	annual_season_precpt text NULL,
	annual_season_temp text NULL,
	annual_temp text NULL,
	antibiotic_regm text NULL,
	antiviral_treatment_agent text NULL,
	api text NULL,
	area_samp_size text NULL,
	association_duration text NULL,
	atmospheric_data text NULL,
	authority text NULL,
	avg_occup text NULL,
	avg_temp text NULL,
	bac_prod text NULL,
	bac_resp text NULL,
	bacteria_carb_prod text NULL,
	bacterial_density text NULL,
	barometric_press text NULL,
	basin_name text NULL,
	benzene text NULL,
	beta_lactamase_family text NULL,
	biochem_oxygen_dem text NULL,
	biocide_used text NULL,
	biol_stat text NULL,
	biomass text NULL,
	bio_material text NULL,
	biomaterial_provider text NULL,
	biospecimen_repository text NULL,
	biospecimen_repository_sample_id text NULL,
	biotic_regm text NULL,
	biotic_relationship text NULL,
	biovar text NULL,
	birth_control text NULL,
	birth_date text NULL,
	birth_location text NULL,
	bishomohopanol text NULL,
	blood_blood_disord text NULL,
	blood_press_diast text NULL,
	blood_press_syst text NULL,
	body_habitat text NULL,
	body_mass_index text NULL,
	body_product text NULL,
	breed text NULL,
	breeding_history text NULL,
	breeding_method text NULL,
	bromide text NULL,
	building_setting text NULL,
	build_occup_type text NULL,
	calcium text NULL,
	carbapenemase text NULL,
	carb_dioxide text NULL,
	carb_nitro_ratio text NULL,
	cell_line text NULL,
	cell_subtype text NULL,
	cell_type text NULL,
	chem_administration text NULL,
	chem_mutagen text NULL,
	chem_oxygen_dem text NULL,
	chem_treatment text NULL,
	child_of text NULL,
	chloride text NULL,
	chlorophyll text NULL,
	climate_environment text NULL,
	clone text NULL,
	clone_lib text NULL,
	collected_by text NULL,
	collection_date text NULL,
	collection_device text NULL,
	collection_method text NULL,
	coll_site_geo_feat text NULL,
	component_organism text NULL,
	compound text NULL,
	conduc text NULL,
	cons_food_stor_dur text NULL,
	cons_food_stor_temp text NULL,
	cons_purch_date text NULL,
	cons_qty_purchased text NULL,
	crop_rotation text NULL,
	crop_yield text NULL,
	cult_isol_date text NULL,
	cultivar text NULL,
	cult_result text NULL,
	cult_result_org text NULL,
	cult_root_med text NULL,
	cult_target text NULL,
	culture_collection text NULL,
	cur_land_use text NULL,
	cur_vegetation text NULL,
	cur_vegetation_meth text NULL,
	date_of_prior_antiviral_treat text NULL,
	date_of_prior_sars_cov_2_infection text NULL,
	date_of_sars_cov_2_vaccination text NULL,
	death_date text NULL,
	density text NULL,
	"depth" text NULL,
	derived_from text NULL,
	dermatology_disord text NULL,
	description text NULL,
	dev_stage text NULL,
	dew_point text NULL,
	diet text NULL,
	dietary_claim_use text NULL,
	diether_lipids text NULL,
	diet_last_six_month text NULL,
	disease text NULL,
	disease_stage text NULL,
	diss_carb_dioxide text NULL,
	diss_hydrogen text NULL,
	diss_inorg_carb text NULL,
	diss_inorg_nitro text NULL,
	diss_inorg_phosp text NULL,
	diss_org_carb text NULL,
	diss_org_nitro text NULL,
	diss_oxygen text NULL,
	dominant_hand text NULL,
	dose text NULL,
	douche text NULL,
	down_par text NULL,
	drainage_class text NULL,
	drug_usage text NULL,
	dry_mass text NULL,
	ecotype text NULL,
	edta_inhibitor_tested text NULL,
	efficiency_percent text NULL,
	elev text NULL,
	emulsions text NULL,
	encoded_traits text NULL,
	enrichment_protocol text NULL,
	env_broad_scale text NULL,
	env_local_scale text NULL,
	env_medium text NULL,
	env_monitoring_zone text NULL,
	env_package text NULL,
	estimated_size text NULL,
	ethnicity text NULL,
	ethylbenzene text NULL,
	experimental_factor text NULL,
	exposure_event text NULL,
	extrachrom_elements text NULL,
	extreme_event text NULL,
	extreme_salinity text NULL,
	extr_weather_event text NULL,
	facility_type text NULL,
	family_id text NULL,
	family_relationship text NULL,
	fao_class text NULL,
	farm_equip text NULL,
	farm_equip_san text NULL,
	farm_equip_shared text NULL,
	farm_water_source text NULL,
	ferm_medium text NULL,
	ferm_ph text NULL,
	ferm_temp text NULL,
	ferm_time text NULL,
	ferm_vessel text NULL,
	fertilizer_admin text NULL,
	fertilizer_regm text NULL,
	field text NULL,
	filter_type text NULL,
	fire text NULL,
	flooding text NULL,
	fluor text NULL,
	foetal_health_stat text NULL,
	food_additive text NULL,
	food_allergen_label text NULL,
	food_contact_surf text NULL,
	food_contain_wrap text NULL,
	food_cooking_proc text NULL,
	food_dis_point text NULL,
	food_dis_point_city text NULL,
	food_harvest_proc text NULL,
	food_industry_class text NULL,
	food_industry_code text NULL,
	food_ingredient text NULL,
	food_name_status text NULL,
	food_origin text NULL,
	food_pack_capacity text NULL,
	food_pack_integrity text NULL,
	food_pack_medium text NULL,
	food_preserv_proc text NULL,
	food_prior_contact text NULL,
	food_processing_method text NULL,
	food_prod text NULL,
	food_prod_char text NULL,
	food_prod_synonym text NULL,
	food_product_qual text NULL,
	food_product_type text NULL,
	food_quality_date text NULL,
	food_source text NULL,
	food_trace_list text NULL,
	food_trav_mode text NULL,
	food_trav_vehic text NULL,
	food_treat_proc text NULL,
	food_type_processed text NULL,
	forma text NULL,
	forma_specialis text NULL,
	freq_clean text NULL,
	fungicide_regm text NULL,
	gap_accession text NULL,
	gap_consent_code text NULL,
	gap_consent_short_name text NULL,
	gap_sample_id text NULL,
	gap_subject_id text NULL,
	gaseous_environment text NULL,
	gaseous_substances text NULL,
	gastrointest_disord text NULL,
	gender_restroom text NULL,
	genetic_mod text NULL,
	genotype text NULL,
	geo_loc_exposure text NULL,
	geo_loc_name text NULL,
	gestation_state text NULL,
	gisaid_accession text NULL,
	gisaid_virus_name text NULL,
	glucosidase_act text NULL,
	gravidity text NULL,
	gravity text NULL,
	growth_facil text NULL,
	growth_habit text NULL,
	growth_hormone_regm text NULL,
	growth_med text NULL,
	growth_medium text NULL,
	growth_protocol text NULL,
	gynecologic_disord text NULL,
	haccp_term text NULL,
	haplotype text NULL,
	hc_produced text NULL,
	hcr text NULL,
	hcr_temp text NULL,
	health_state text NULL,
	heat_cool_type text NULL,
	heavy_metals text NULL,
	heavy_metals_meth text NULL,
	height_or_length text NULL,
	herbicide_regm text NULL,
	histological_type text NULL,
	hiv_stat text NULL,
	horizon_meth text NULL,
	host text NULL,
	host_age text NULL,
	host_am text NULL,
	host_anatomical_material text NULL,
	host_anatomical_part text NULL,
	host_animal_breed text NULL,
	host_blood_press_diast text NULL,
	host_blood_press_syst text NULL,
	host_body_habitat text NULL,
	host_body_mass_index text NULL,
	host_body_product text NULL,
	host_body_temp text NULL,
	host_cellular_loc text NULL,
	host_color text NULL,
	host_common_name text NULL,
	host_dependence text NULL,
	host_description text NULL,
	host_diet text NULL,
	host_disease text NULL,
	host_disease_outcome text NULL,
	host_disease_stage text NULL,
	host_dry_mass text NULL,
	host_family_relationship text NULL,
	host_genotype text NULL,
	host_group_size text NULL,
	host_growth_cond text NULL,
	host_health_state text NULL,
	host_height text NULL,
	host_hiv_stat text NULL,
	host_housing text NULL,
	host_infra_specific_name text NULL,
	host_infra_specific_rank text NULL,
	host_last_meal text NULL,
	host_length text NULL,
	host_life_stage text NULL,
	host_number text NULL,
	host_occupation text NULL,
	host_of_host_sub_id text NULL,
	host_phenotype text NULL,
	host_pulse text NULL,
	host_recent_travel_loc text NULL,
	host_recent_travel_return_date text NULL,
	host_sex text NULL,
	host_shape text NULL,
	host_specificity text NULL,
	host_specimen_voucher text NULL,
	host_subject_id text NULL,
	host_subspecf_genlin text NULL,
	host_substrate text NULL,
	host_symbiont text NULL,
	host_taxid text NULL,
	host_tissue_sampled text NULL,
	host_tot_mass text NULL,
	host_variety text NULL,
	host_wet_mass text NULL,
	hrt text NULL,
	humidity text NULL,
	humidity_regm text NULL,
	hygienic_area text NULL,
	hysterectomy text NULL,
	identified_by text NULL,
	ifsac_category text NULL,
	ihmc_medication_code text NULL,
	image_file text NULL,
	indoor_space text NULL,
	indoor_surf text NULL,
	indoor_surf_subpart text NULL,
	indust_eff_percent text NULL,
	infra_specific_name text NULL,
	infra_specific_rank text NULL,
	inorg_particles text NULL,
	intended_consumer text NULL,
	investigation_type text NULL,
	isolate text NULL,
	isolate_name_alias text NULL,
	isolation_source text NULL,
	isol_growth_condt text NULL,
	is_tumor text NULL,
	iwf text NULL,
	karyotype text NULL,
	kidney_disord text NULL,
	"label" text NULL,
	label_claims text NULL,
	lab_host text NULL,
	last_meal text NULL,
	lat_lon text NULL,
	life_stage text NULL,
	light_intensity text NULL,
	light_regm text NULL,
	light_type text NULL,
	link_addit_analys text NULL,
	link_class_info text NULL,
	link_climate_info text NULL,
	lithology text NULL,
	liver_disord text NULL,
	local_class text NULL,
	local_class_meth text NULL,
	location_in_facility text NULL,
	lot_number text NULL,
	magnesium text NULL,
	material_condition text NULL,
	maternal_health_stat text NULL,
	mating_type text NULL,
	mean_frict_vel text NULL,
	mean_peak_frict_vel text NULL,
	mechanical_damage text NULL,
	medic_hist_perform text NULL,
	menarche text NULL,
	menopause text NULL,
	metagenome_source text NULL,
	methane text NULL,
	microb_cult_med text NULL,
	microbial_biomass text NULL,
	microbial_biomass_meth text NULL,
	mineral_nutr_regm text NULL,
	misc_param text NULL,
	mode_transmission text NULL,
	molecular_data_type text NULL,
	morphology text NULL,
	n_alkanes text NULL,
	narms_isolate_number text NULL,
	neg_cont_type text NULL,
	nitrate text NULL,
	nitrite text NULL,
	nitro text NULL,
	non_mineral_nutr_regm text NULL,
	nose_mouth_teeth_throat_disord text NULL,
	nose_throat_disord text NULL,
	number_pets text NULL,
	num_replicons text NULL,
	num_samp_collect text NULL,
	occupant_dens_samp text NULL,
	occupation text NULL,
	occup_samp text NULL,
	omics_observ_id text NULL,
	organism_count text NULL,
	org_carb text NULL,
	org_matter text NULL,
	orgmod_note text NULL,
	org_nitro text NULL,
	org_particles text NULL,
	original_subject_id text NULL,
	outbreak text NULL,
	oxygen text NULL,
	oxy_stat_samp text NULL,
	particle_class text NULL,
	part_org_carb text NULL,
	part_org_nitro text NULL,
	part_plant_animal text NULL,
	passage_history text NULL,
	passage_method text NULL,
	passage_number text NULL,
	pathogenicity text NULL,
	pathotype text NULL,
	pathovar text NULL,
	permeability text NULL,
	perturbation text NULL,
	pesticide_regm text NULL,
	pet_farm_animal text NULL,
	petroleum_hydrocarb text NULL,
	ph text NULL,
	phaeopigments text NULL,
	phenotype text NULL,
	ph_meth text NULL,
	phosphate text NULL,
	phosplipid_fatt_acid text NULL,
	photon_flux text NULL,
	photosynt_activ text NULL,
	photosynt_activ_meth text NULL,
	ph_regm text NULL,
	plant_body_site text NULL,
	plant_growth_med text NULL,
	plant_product text NULL,
	plant_sex text NULL,
	plant_struc text NULL,
	plant_water_method text NULL,
	ploidy text NULL,
	pollutants text NULL,
	pool_dna_extracts text NULL,
	population text NULL,
	population_description text NULL,
	porosity text NULL,
	pos_cont_type text NULL,
	potassium text NULL,
	pregnancy text NULL,
	pres_animal_insect text NULL,
	pressure text NULL,
	pre_treatment text NULL,
	previous_land_use text NULL,
	previous_land_use_meth text NULL,
	primary_prod text NULL,
	primary_treatment text NULL,
	prior_sars_cov_2_antiviral_treat text NULL,
	prior_sars_cov_2_infection text NULL,
	prior_sars_cov_2_vaccination text NULL,
	prod_label_claims text NULL,
	profile_position text NULL,
	project_name text NULL,
	propagation text NULL,
	pulmonary_disord text NULL,
	pulse text NULL,
	purpose_of_sampling text NULL,
	purpose_of_sequencing text NULL,
	purpose_of_ww_sampling text NULL,
	purpose_of_ww_sequencing text NULL,
	race text NULL,
	radiation_regm text NULL,
	rainfall_regm text NULL,
	reactor_type text NULL,
	redox_potential text NULL,
	ref_biomaterial text NULL,
	reference_material text NULL,
	rel_air_humidity text NULL,
	rel_location text NULL,
	rel_to_oxygen text NULL,
	repository text NULL,
	risk_group text NULL,
	room_type text NULL,
	root_cond text NULL,
	root_med_carbon text NULL,
	root_med_macronutr text NULL,
	root_med_micronutr text NULL,
	root_med_ph text NULL,
	root_med_regl text NULL,
	root_med_solid text NULL,
	root_med_suppl text NULL,
	route_transmission text NULL,
	salinity text NULL,
	salinity_meth text NULL,
	salt_regm text NULL,
	same_as text NULL,
	samp_capt_status text NULL,
	samp_collect_device text NULL,
	samp_collect_method text NULL,
	samp_collect_point text NULL,
	samp_dis_stage text NULL,
	sample_name text NULL,
	sample_type text NULL,
	samp_mat_process text NULL,
	samp_mat_type text NULL,
	samp_pooling text NULL,
	samp_rep_biol text NULL,
	samp_rep_tech text NULL,
	samp_salinity text NULL,
	samp_size text NULL,
	samp_sort_meth text NULL,
	samp_source_mat_cat text NULL,
	samp_stor_device text NULL,
	samp_store_dur text NULL,
	samp_store_loc text NULL,
	samp_store_sol text NULL,
	samp_store_temp text NULL,
	samp_stor_media text NULL,
	samp_subtype text NULL,
	samp_transport_temp text NULL,
	samp_vol_we_dna_ext text NULL,
	sars_cov_2_diag_gene_name_1 text NULL,
	sars_cov_2_diag_gene_name_2 text NULL,
	sars_cov_2_diag_pcr_ct_value_1 text NULL,
	sars_cov_2_diag_pcr_ct_value_2 text NULL,
	season text NULL,
	season_environment text NULL,
	season_humidity text NULL,
	season_precpt text NULL,
	season_temp text NULL,
	secondary_treatment text NULL,
	sediment_type text NULL,
	sequenced_by text NULL,
	serogroup text NULL,
	serotype text NULL,
	serovar text NULL,
	sewage_type text NULL,
	sex text NULL,
	sexual_act text NULL,
	sieving text NULL,
	silicate text NULL,
	size_frac text NULL,
	size_frac_low text NULL,
	size_frac_up text NULL,
	slope_aspect text NULL,
	slope_gradient text NULL,
	sludge_retent_time text NULL,
	smoker text NULL,
	sodium text NULL,
	soil_conductivity text NULL,
	soil_cover text NULL,
	soil_depth text NULL,
	soil_horizon text NULL,
	soil_ph text NULL,
	soil_temp text NULL,
	soil_text_measure text NULL,
	soil_texture_class text NULL,
	soil_texture_meth text NULL,
	soil_type text NULL,
	soil_type_meth text NULL,
	solar_irradiance text NULL,
	soluble_inorg_mat text NULL,
	soluble_org_mat text NULL,
	soluble_react_phosp text NULL,
	source_material_id text NULL,
	source_name text NULL,
	source_type text NULL,
	source_uvig text NULL,
	space_typ_state text NULL,
	special_diet text NULL,
	"specific" text NULL,
	specimen_voucher text NULL,
	spec_intended_cons text NULL,
	spikein_count text NULL,
	spikein_org text NULL,
	spikein_strain text NULL,
	standing_water_regm text NULL,
	store_cond text NULL,
	strain text NULL,
	stud_book_number text NULL,
	study_complt_stat text NULL,
	study_design text NULL,
	study_disease text NULL,
	study_inc_dur text NULL,
	study_inc_temp text NULL,
	study_name text NULL,
	study_tmnt text NULL,
	subclone text NULL,
	subgroup text NULL,
	subject_is_affected text NULL,
	submitted_sample_id text NULL,
	submitted_subject_id text NULL,
	submitter_handle text NULL,
	subspecf_gen_lin text NULL,
	sub_species text NULL,
	subsrc_note text NULL,
	substrain text NULL,
	substrate text NULL,
	substructure_type text NULL,
	subtype text NULL,
	sulfate text NULL,
	sulfate_fw text NULL,
	sulfide text NULL,
	super_population_code text NULL,
	super_population_description text NULL,
	surface_orientation text NULL,
	surf_air_cont text NULL,
	surf_humidity text NULL,
	surf_material text NULL,
	surf_moisture text NULL,
	surf_moisture_ph text NULL,
	surf_temp text NULL,
	suspend_part_matter text NULL,
	suspend_solids text NULL,
	sym_life_cycle_type text NULL,
	tan text NULL,
	teleomorph text NULL,
	"temp" text NULL,
	tertiary_treatment text NULL,
	tidal_stage text NULL,
	tillage text NULL,
	time_last_toothbrush text NULL,
	timepoint text NULL,
	time_since_last_wash text NULL,
	tiss_cult_growth_med text NULL,
	tissue text NULL,
	tissue_lib text NULL,
	toluene text NULL,
	tot_carb text NULL,
	tot_depth_water_col text NULL,
	tot_diss_nitro text NULL,
	tot_inorg_nitro text NULL,
	tot_iron text NULL,
	tot_mass text NULL,
	tot_nitro text NULL,
	tot_nitro_cont_meth text NULL,
	tot_n_meth text NULL,
	tot_org_carb text NULL,
	tot_org_c_meth text NULL,
	tot_part_carb text NULL,
	tot_phosp text NULL,
	tot_phosphate text NULL,
	tot_sulfur text NULL,
	train_line text NULL,
	travel_out_six_month text NULL,
	treatment text NULL,
	trophic_level text NULL,
	turbidity text NULL,
	twin_sibling text NULL,
	type_of_symbiosis text NULL,
	type_status text NULL,
	type_strain text NULL,
	typ_occupant_dens text NULL,
	upstream_intervention text NULL,
	urine_collect_meth text NULL,
	urogenit_disord text NULL,
	urogenit_tract_disor text NULL,
	vaccine_received text NULL,
	variety text NULL,
	ventilation_rate text NULL,
	ventilation_type text NULL,
	vfa text NULL,
	vfa_fw text NULL,
	virus_enrich_appr text NULL,
	virus_isolate_of_prior_infection text NULL,
	volatile_org_comp text NULL,
	wastewater_type text NULL,
	water_content text NULL,
	water_content_soil text NULL,
	water_content_soil_meth text NULL,
	water_current text NULL,
	water_cut text NULL,
	watering_regm text NULL,
	water_ph text NULL,
	water_source_shared text NULL,
	water_temp_regm text NULL,
	weekday text NULL,
	weight_loss_3_month text NULL,
	wet_mass text NULL,
	wga_amp_appr text NULL,
	wind_direction text NULL,
	window_open_freq text NULL,
	wind_speed text NULL,
	ww_endog_control_1 text NULL,
	ww_endog_control_1_conc text NULL,
	ww_endog_control_1_protocol text NULL,
	ww_endog_control_1_units text NULL,
	ww_endog_control_2 text NULL,
	ww_endog_control_2_conc text NULL,
	ww_endog_control_2_protocol text NULL,
	ww_endog_control_2_units text NULL,
	ww_flow text NULL,
	ww_industrial_effluent_percent text NULL,
	ww_ph text NULL,
	ww_population text NULL,
	ww_population_source text NULL,
	ww_pre_treatment text NULL,
	ww_primary_sludge_retention_time text NULL,
	ww_processing_protocol text NULL,
	ww_sample_duration text NULL,
	ww_sample_matrix text NULL,
	ww_sample_salinity text NULL,
	ww_sample_site text NULL,
	ww_sample_type text NULL,
	ww_surv_jurisdiction text NULL,
	ww_surv_system_sample_id text NULL,
	ww_surv_target_1 text NULL,
	ww_surv_target_1_conc text NULL,
	ww_surv_target_1_conc_unit text NULL,
	ww_surv_target_1_extract text NULL,
	ww_surv_target_1_extract_unit text NULL,
	ww_surv_target_1_gene text NULL,
	ww_surv_target_1_known_present text NULL,
	ww_surv_target_1_protocol text NULL,
	ww_surv_target_2 text NULL,
	ww_surv_target_2_conc text NULL,
	ww_surv_target_2_conc_unit text NULL,
	ww_surv_target_2_extract text NULL,
	ww_surv_target_2_extract_unit text NULL,
	ww_surv_target_2_gene text NULL,
	ww_surv_target_2_known_present text NULL,
	ww_surv_target_2_protocol text NULL,
	ww_temperature text NULL,
	ww_total_suspended_solids text NULL,
	CONSTRAINT ncbi_attributes_harmonized_wide_pkey PRIMARY KEY (raw_id)
);


-- public.ncbi_metagenomes definition

-- Drop table

-- DROP TABLE public.ncbi_metagenomes;

CREATE TABLE public.ncbi_metagenomes (
	taxon_id int4 NOT NULL,
	description text NULL,
	CONSTRAINT ncbi_metagenomes_pkey PRIMARY KEY (taxon_id)
);
CREATE INDEX ncbi_metagenomes_taxon_id_idx ON public.ncbi_metagenomes USING btree (taxon_id);


-- public.ncbi_mixs_class_mappings definition

-- Drop table

-- DROP TABLE public.ncbi_mixs_class_mappings;

CREATE TABLE public.ncbi_mixs_class_mappings (
	ncbi text NULL,
	mixs text NULL
);
CREATE INDEX ncbi_mixs_class_mappings_ncbi_idx ON public.ncbi_mixs_class_mappings USING btree (ncbi, mixs);


-- public.ncbi_package_attribute_use definition

-- Drop table

-- DROP TABLE public.ncbi_package_attribute_use;

CREATE TABLE public.ncbi_package_attribute_use (
	attribute_harmonized_name text NULL,
	package text NULL,
	use text NULL
);
CREATE INDEX ncbi_package_attribute_use_attribute_harmonized_name_idx ON public.ncbi_package_attribute_use USING btree (attribute_harmonized_name, package, use);


-- public.ncbi_package_definitions definition

-- Drop table

-- DROP TABLE public.ncbi_package_definitions;

CREATE TABLE public.ncbi_package_definitions (
	"Name" text NULL,
	"DisplayName" text NULL,
	"EnvPackage" text NULL,
	"EnvPackageDisplay" text NULL,
	"Example" text NULL,
	"NotAppropriateFor" text NULL,
	"ShortName" text NULL,
	"group" text NULL,
	"Description" text NULL,
	"attributes" text NULL
);
CREATE INDEX ncbi_package_definitions_name_idx ON public.ncbi_package_definitions USING btree ("Name", "DisplayName", "EnvPackage", "EnvPackageDisplay", "ShortName", "group");


-- public.non_attribute_metadata definition

-- Drop table

-- DROP TABLE public.non_attribute_metadata;

CREATE TABLE public.non_attribute_metadata (
	raw_id int8 NOT NULL,
	accession text NOT NULL,
	bp_id text NULL,
	contributors text NOT NULL,
	model text NOT NULL,
	owner_abbreviation text NULL,
	owner_text text NULL,
	owner_url text NULL,
	package text NOT NULL,
	package_name text NOT NULL,
	paragraph text NULL,
	prefixed_id text NOT NULL,
	primary_id text NOT NULL,
	samp_name text NULL,
	sra_id text NULL,
	status text NULL,
	status_date text NULL,
	synonym text NULL,
	table_caption text NULL,
	taxonomy_id text NULL,
	taxonomy_name text NULL,
	title text NULL
);
CREATE UNIQUE INDEX non_attribute_metadata_accession_idx ON public.non_attribute_metadata USING btree (accession);
CREATE INDEX non_attribute_metadata_bp_id_idx ON public.non_attribute_metadata USING btree (bp_id);
CREATE UNIQUE INDEX non_attribute_metadata_prefixed_id_idx ON public.non_attribute_metadata USING btree (prefixed_id);
CREATE UNIQUE INDEX non_attribute_metadata_primary_id_idx ON public.non_attribute_metadata USING btree (primary_id);
CREATE UNIQUE INDEX non_attribute_metadata_raw_id_idx ON public.non_attribute_metadata USING btree (raw_id);


-- public.prefix definition

-- Drop table

-- DROP TABLE public.prefix;

CREATE TABLE public.prefix (
	prefix text NULL,
	base text NULL
);


-- public.statements definition

-- Drop table

-- DROP TABLE public.statements;

CREATE TABLE public.statements (
	stanza text NULL,
	subject text NULL,
	predicate text NULL,
	"object" text NULL,
	value text NULL,
	"datatype" text NULL,
	"language" text NULL,
	graph text NULL
);
CREATE INDEX idx_190398_statements_p ON public.statements USING btree (predicate);
CREATE INDEX idx_190398_statements_spo ON public.statements USING btree (subject, predicate, object);
CREATE INDEX idx_190398_statements_spv ON public.statements USING btree (subject, predicate, value);


-- public.attributes_plus source

CREATE OR REPLACE VIEW public.attributes_plus
AS SELECT nam.accession,
    nam.bp_id,
    nam.contributors,
    nam.model,
    nam.owner_abbreviation,
    nam.owner_text,
    nam.owner_url,
    nam.package,
    nam.package_name,
    nam.paragraph,
    nam.prefixed_id,
    nam.primary_id,
    nam.samp_name,
    nam.sra_id,
    nam.status,
    nam.status_date,
    nam.synonym,
    nam.table_caption,
    nam.taxonomy_id,
    nam.taxonomy_name,
    nam.title,
    nahw.raw_id,
    nahw.abs_air_humidity,
    nahw.additional_info,
    nahw.add_recov_method,
    nahw.address,
    nahw.adjacent_environment,
    nahw.adj_room,
    nahw.affection_status,
    nahw.age,
    nahw.agrochem_addition,
    nahw.air_pm_concen,
    nahw.air_temp,
    nahw.air_temp_regm,
    nahw.alkalinity,
    nahw.alkalinity_method,
    nahw.alkyl_diethers,
    nahw.al_sat,
    nahw.al_sat_meth,
    nahw.altitude,
    nahw.aminopept_act,
    nahw.ammonium,
    nahw.amniotic_fluid_color,
    nahw.analyte_type,
    nahw.anamorph,
    nahw.ances_data,
    nahw.animal_body_cond,
    nahw.animal_diet,
    nahw.animal_env,
    nahw.animal_intrusion,
    nahw.animal_sex,
    nahw.anim_water_method,
    nahw.annual_precpt,
    nahw.annual_season_precpt,
    nahw.annual_season_temp,
    nahw.annual_temp,
    nahw.antibiotic_regm,
    nahw.antiviral_treatment_agent,
    nahw.api,
    nahw.area_samp_size,
    nahw.association_duration,
    nahw.atmospheric_data,
    nahw.authority,
    nahw.avg_occup,
    nahw.avg_temp,
    nahw.bac_prod,
    nahw.bac_resp,
    nahw.bacteria_carb_prod,
    nahw.bacterial_density,
    nahw.barometric_press,
    nahw.basin_name,
    nahw.benzene,
    nahw.beta_lactamase_family,
    nahw.biochem_oxygen_dem,
    nahw.biocide_used,
    nahw.biol_stat,
    nahw.biomass,
    nahw.bio_material,
    nahw.biomaterial_provider,
    nahw.biospecimen_repository,
    nahw.biospecimen_repository_sample_id,
    nahw.biotic_regm,
    nahw.biotic_relationship,
    nahw.biovar,
    nahw.birth_control,
    nahw.birth_date,
    nahw.birth_location,
    nahw.bishomohopanol,
    nahw.blood_blood_disord,
    nahw.blood_press_diast,
    nahw.blood_press_syst,
    nahw.body_habitat,
    nahw.body_mass_index,
    nahw.body_product,
    nahw.breed,
    nahw.breeding_history,
    nahw.breeding_method,
    nahw.bromide,
    nahw.building_setting,
    nahw.build_occup_type,
    nahw.calcium,
    nahw.carbapenemase,
    nahw.carb_dioxide,
    nahw.carb_nitro_ratio,
    nahw.cell_line,
    nahw.cell_subtype,
    nahw.cell_type,
    nahw.chem_administration,
    nahw.chem_mutagen,
    nahw.chem_oxygen_dem,
    nahw.chem_treatment,
    nahw.child_of,
    nahw.chloride,
    nahw.chlorophyll,
    nahw.climate_environment,
    nahw.clone,
    nahw.clone_lib,
    nahw.collected_by,
    nahw.collection_date,
    nahw.collection_device,
    nahw.collection_method,
    nahw.coll_site_geo_feat,
    nahw.component_organism,
    nahw.compound,
    nahw.conduc,
    nahw.cons_food_stor_dur,
    nahw.cons_food_stor_temp,
    nahw.cons_purch_date,
    nahw.cons_qty_purchased,
    nahw.crop_rotation,
    nahw.crop_yield,
    nahw.cult_isol_date,
    nahw.cultivar,
    nahw.cult_result,
    nahw.cult_result_org,
    nahw.cult_root_med,
    nahw.cult_target,
    nahw.culture_collection,
    nahw.cur_land_use,
    nahw.cur_vegetation,
    nahw.cur_vegetation_meth,
    nahw.date_of_prior_antiviral_treat,
    nahw.date_of_prior_sars_cov_2_infection,
    nahw.date_of_sars_cov_2_vaccination,
    nahw.death_date,
    nahw.density,
    nahw.depth,
    nahw.derived_from,
    nahw.dermatology_disord,
    nahw.description,
    nahw.dev_stage,
    nahw.dew_point,
    nahw.diet,
    nahw.dietary_claim_use,
    nahw.diether_lipids,
    nahw.diet_last_six_month,
    nahw.disease,
    nahw.disease_stage,
    nahw.diss_carb_dioxide,
    nahw.diss_hydrogen,
    nahw.diss_inorg_carb,
    nahw.diss_inorg_nitro,
    nahw.diss_inorg_phosp,
    nahw.diss_org_carb,
    nahw.diss_org_nitro,
    nahw.diss_oxygen,
    nahw.dominant_hand,
    nahw.dose,
    nahw.douche,
    nahw.down_par,
    nahw.drainage_class,
    nahw.drug_usage,
    nahw.dry_mass,
    nahw.ecotype,
    nahw.edta_inhibitor_tested,
    nahw.efficiency_percent,
    nahw.elev,
    nahw.emulsions,
    nahw.encoded_traits,
    nahw.enrichment_protocol,
    nahw.env_broad_scale,
    nahw.env_local_scale,
    nahw.env_medium,
    nahw.env_monitoring_zone,
    nahw.env_package,
    nahw.estimated_size,
    nahw.ethnicity,
    nahw.ethylbenzene,
    nahw.experimental_factor,
    nahw.exposure_event,
    nahw.extrachrom_elements,
    nahw.extreme_event,
    nahw.extreme_salinity,
    nahw.extr_weather_event,
    nahw.facility_type,
    nahw.family_id,
    nahw.family_relationship,
    nahw.fao_class,
    nahw.farm_equip,
    nahw.farm_equip_san,
    nahw.farm_equip_shared,
    nahw.farm_water_source,
    nahw.ferm_medium,
    nahw.ferm_ph,
    nahw.ferm_temp,
    nahw.ferm_time,
    nahw.ferm_vessel,
    nahw.fertilizer_admin,
    nahw.fertilizer_regm,
    nahw.field,
    nahw.filter_type,
    nahw.fire,
    nahw.flooding,
    nahw.fluor,
    nahw.foetal_health_stat,
    nahw.food_additive,
    nahw.food_allergen_label,
    nahw.food_contact_surf,
    nahw.food_contain_wrap,
    nahw.food_cooking_proc,
    nahw.food_dis_point,
    nahw.food_dis_point_city,
    nahw.food_harvest_proc,
    nahw.food_industry_class,
    nahw.food_industry_code,
    nahw.food_ingredient,
    nahw.food_name_status,
    nahw.food_origin,
    nahw.food_pack_capacity,
    nahw.food_pack_integrity,
    nahw.food_pack_medium,
    nahw.food_preserv_proc,
    nahw.food_prior_contact,
    nahw.food_processing_method,
    nahw.food_prod,
    nahw.food_prod_char,
    nahw.food_prod_synonym,
    nahw.food_product_qual,
    nahw.food_product_type,
    nahw.food_quality_date,
    nahw.food_source,
    nahw.food_trace_list,
    nahw.food_trav_mode,
    nahw.food_trav_vehic,
    nahw.food_treat_proc,
    nahw.food_type_processed,
    nahw.forma,
    nahw.forma_specialis,
    nahw.freq_clean,
    nahw.fungicide_regm,
    nahw.gap_accession,
    nahw.gap_consent_code,
    nahw.gap_consent_short_name,
    nahw.gap_sample_id,
    nahw.gap_subject_id,
    nahw.gaseous_environment,
    nahw.gaseous_substances,
    nahw.gastrointest_disord,
    nahw.gender_restroom,
    nahw.genetic_mod,
    nahw.genotype,
    nahw.geo_loc_exposure,
    nahw.geo_loc_name,
    nahw.gestation_state,
    nahw.gisaid_accession,
    nahw.gisaid_virus_name,
    nahw.glucosidase_act,
    nahw.gravidity,
    nahw.gravity,
    nahw.growth_facil,
    nahw.growth_habit,
    nahw.growth_hormone_regm,
    nahw.growth_med,
    nahw.growth_medium,
    nahw.growth_protocol,
    nahw.gynecologic_disord,
    nahw.haccp_term,
    nahw.haplotype,
    nahw.hc_produced,
    nahw.hcr,
    nahw.hcr_temp,
    nahw.health_state,
    nahw.heat_cool_type,
    nahw.heavy_metals,
    nahw.heavy_metals_meth,
    nahw.height_or_length,
    nahw.herbicide_regm,
    nahw.histological_type,
    nahw.hiv_stat,
    nahw.horizon_meth,
    nahw.host,
    nahw.host_age,
    nahw.host_am,
    nahw.host_anatomical_material,
    nahw.host_anatomical_part,
    nahw.host_animal_breed,
    nahw.host_blood_press_diast,
    nahw.host_blood_press_syst,
    nahw.host_body_habitat,
    nahw.host_body_mass_index,
    nahw.host_body_product,
    nahw.host_body_temp,
    nahw.host_cellular_loc,
    nahw.host_color,
    nahw.host_common_name,
    nahw.host_dependence,
    nahw.host_description,
    nahw.host_diet,
    nahw.host_disease,
    nahw.host_disease_outcome,
    nahw.host_disease_stage,
    nahw.host_dry_mass,
    nahw.host_family_relationship,
    nahw.host_genotype,
    nahw.host_group_size,
    nahw.host_growth_cond,
    nahw.host_health_state,
    nahw.host_height,
    nahw.host_hiv_stat,
    nahw.host_housing,
    nahw.host_infra_specific_name,
    nahw.host_infra_specific_rank,
    nahw.host_last_meal,
    nahw.host_length,
    nahw.host_life_stage,
    nahw.host_number,
    nahw.host_occupation,
    nahw.host_of_host_sub_id,
    nahw.host_phenotype,
    nahw.host_pulse,
    nahw.host_recent_travel_loc,
    nahw.host_recent_travel_return_date,
    nahw.host_sex,
    nahw.host_shape,
    nahw.host_specificity,
    nahw.host_specimen_voucher,
    nahw.host_subject_id,
    nahw.host_subspecf_genlin,
    nahw.host_substrate,
    nahw.host_symbiont,
    nahw.host_taxid,
    nahw.host_tissue_sampled,
    nahw.host_tot_mass,
    nahw.host_variety,
    nahw.host_wet_mass,
    nahw.hrt,
    nahw.humidity,
    nahw.humidity_regm,
    nahw.hygienic_area,
    nahw.hysterectomy,
    nahw.identified_by,
    nahw.ifsac_category,
    nahw.ihmc_medication_code,
    nahw.image_file,
    nahw.indoor_space,
    nahw.indoor_surf,
    nahw.indoor_surf_subpart,
    nahw.indust_eff_percent,
    nahw.infra_specific_name,
    nahw.infra_specific_rank,
    nahw.inorg_particles,
    nahw.intended_consumer,
    nahw.investigation_type,
    nahw.isolate,
    nahw.isolate_name_alias,
    nahw.isolation_source,
    nahw.isol_growth_condt,
    nahw.is_tumor,
    nahw.iwf,
    nahw.karyotype,
    nahw.kidney_disord,
    nahw.label,
    nahw.label_claims,
    nahw.lab_host,
    nahw.last_meal,
    nahw.lat_lon,
    nahw.life_stage,
    nahw.light_intensity,
    nahw.light_regm,
    nahw.light_type,
    nahw.link_addit_analys,
    nahw.link_class_info,
    nahw.link_climate_info,
    nahw.lithology,
    nahw.liver_disord,
    nahw.local_class,
    nahw.local_class_meth,
    nahw.location_in_facility,
    nahw.lot_number,
    nahw.magnesium,
    nahw.material_condition,
    nahw.maternal_health_stat,
    nahw.mating_type,
    nahw.mean_frict_vel,
    nahw.mean_peak_frict_vel,
    nahw.mechanical_damage,
    nahw.medic_hist_perform,
    nahw.menarche,
    nahw.menopause,
    nahw.metagenome_source,
    nahw.methane,
    nahw.microb_cult_med,
    nahw.microbial_biomass,
    nahw.microbial_biomass_meth,
    nahw.mineral_nutr_regm,
    nahw.misc_param,
    nahw.mode_transmission,
    nahw.molecular_data_type,
    nahw.morphology,
    nahw.n_alkanes,
    nahw.narms_isolate_number,
    nahw.neg_cont_type,
    nahw.nitrate,
    nahw.nitrite,
    nahw.nitro,
    nahw.non_mineral_nutr_regm,
    nahw.nose_mouth_teeth_throat_disord,
    nahw.nose_throat_disord,
    nahw.number_pets,
    nahw.num_replicons,
    nahw.num_samp_collect,
    nahw.occupant_dens_samp,
    nahw.occupation,
    nahw.occup_samp,
    nahw.omics_observ_id,
    nahw.organism_count,
    nahw.org_carb,
    nahw.org_matter,
    nahw.orgmod_note,
    nahw.org_nitro,
    nahw.org_particles,
    nahw.original_subject_id,
    nahw.outbreak,
    nahw.oxygen,
    nahw.oxy_stat_samp,
    nahw.particle_class,
    nahw.part_org_carb,
    nahw.part_org_nitro,
    nahw.part_plant_animal,
    nahw.passage_history,
    nahw.passage_method,
    nahw.passage_number,
    nahw.pathogenicity,
    nahw.pathotype,
    nahw.pathovar,
    nahw.permeability,
    nahw.perturbation,
    nahw.pesticide_regm,
    nahw.pet_farm_animal,
    nahw.petroleum_hydrocarb,
    nahw.ph,
    nahw.phaeopigments,
    nahw.phenotype,
    nahw.ph_meth,
    nahw.phosphate,
    nahw.phosplipid_fatt_acid,
    nahw.photon_flux,
    nahw.photosynt_activ,
    nahw.photosynt_activ_meth,
    nahw.ph_regm,
    nahw.plant_body_site,
    nahw.plant_growth_med,
    nahw.plant_product,
    nahw.plant_sex,
    nahw.plant_struc,
    nahw.plant_water_method,
    nahw.ploidy,
    nahw.pollutants,
    nahw.pool_dna_extracts,
    nahw.population,
    nahw.population_description,
    nahw.porosity,
    nahw.pos_cont_type,
    nahw.potassium,
    nahw.pregnancy,
    nahw.pres_animal_insect,
    nahw.pressure,
    nahw.pre_treatment,
    nahw.previous_land_use,
    nahw.previous_land_use_meth,
    nahw.primary_prod,
    nahw.primary_treatment,
    nahw.prior_sars_cov_2_antiviral_treat,
    nahw.prior_sars_cov_2_infection,
    nahw.prior_sars_cov_2_vaccination,
    nahw.prod_label_claims,
    nahw.profile_position,
    nahw.project_name,
    nahw.propagation,
    nahw.pulmonary_disord,
    nahw.pulse,
    nahw.purpose_of_sampling,
    nahw.purpose_of_sequencing,
    nahw.purpose_of_ww_sampling,
    nahw.purpose_of_ww_sequencing,
    nahw.race,
    nahw.radiation_regm,
    nahw.rainfall_regm,
    nahw.reactor_type,
    nahw.redox_potential,
    nahw.ref_biomaterial,
    nahw.reference_material,
    nahw.rel_air_humidity,
    nahw.rel_location,
    nahw.rel_to_oxygen,
    nahw.repository,
    nahw.risk_group,
    nahw.room_type,
    nahw.root_cond,
    nahw.root_med_carbon,
    nahw.root_med_macronutr,
    nahw.root_med_micronutr,
    nahw.root_med_ph,
    nahw.root_med_regl,
    nahw.root_med_solid,
    nahw.root_med_suppl,
    nahw.route_transmission,
    nahw.salinity,
    nahw.salinity_meth,
    nahw.salt_regm,
    nahw.same_as,
    nahw.samp_capt_status,
    nahw.samp_collect_device,
    nahw.samp_collect_method,
    nahw.samp_collect_point,
    nahw.samp_dis_stage,
    nahw.sample_name,
    nahw.sample_type,
    nahw.samp_mat_process,
    nahw.samp_mat_type,
    nahw.samp_pooling,
    nahw.samp_rep_biol,
    nahw.samp_rep_tech,
    nahw.samp_salinity,
    nahw.samp_size,
    nahw.samp_sort_meth,
    nahw.samp_source_mat_cat,
    nahw.samp_stor_device,
    nahw.samp_store_dur,
    nahw.samp_store_loc,
    nahw.samp_store_sol,
    nahw.samp_store_temp,
    nahw.samp_stor_media,
    nahw.samp_subtype,
    nahw.samp_transport_temp,
    nahw.samp_vol_we_dna_ext,
    nahw.sars_cov_2_diag_gene_name_1,
    nahw.sars_cov_2_diag_gene_name_2,
    nahw.sars_cov_2_diag_pcr_ct_value_1,
    nahw.sars_cov_2_diag_pcr_ct_value_2,
    nahw.season,
    nahw.season_environment,
    nahw.season_humidity,
    nahw.season_precpt,
    nahw.season_temp,
    nahw.secondary_treatment,
    nahw.sediment_type,
    nahw.sequenced_by,
    nahw.serogroup,
    nahw.serotype,
    nahw.serovar,
    nahw.sewage_type,
    nahw.sex,
    nahw.sexual_act,
    nahw.sieving,
    nahw.silicate,
    nahw.size_frac,
    nahw.size_frac_low,
    nahw.size_frac_up,
    nahw.slope_aspect,
    nahw.slope_gradient,
    nahw.sludge_retent_time,
    nahw.smoker,
    nahw.sodium,
    nahw.soil_conductivity,
    nahw.soil_cover,
    nahw.soil_depth,
    nahw.soil_horizon,
    nahw.soil_ph,
    nahw.soil_temp,
    nahw.soil_text_measure,
    nahw.soil_texture_class,
    nahw.soil_texture_meth,
    nahw.soil_type,
    nahw.soil_type_meth,
    nahw.solar_irradiance,
    nahw.soluble_inorg_mat,
    nahw.soluble_org_mat,
    nahw.soluble_react_phosp,
    nahw.source_material_id,
    nahw.source_name,
    nahw.source_type,
    nahw.source_uvig,
    nahw.space_typ_state,
    nahw.special_diet,
    nahw.specific,
    nahw.specimen_voucher,
    nahw.spec_intended_cons,
    nahw.spikein_count,
    nahw.spikein_org,
    nahw.spikein_strain,
    nahw.standing_water_regm,
    nahw.store_cond,
    nahw.strain,
    nahw.stud_book_number,
    nahw.study_complt_stat,
    nahw.study_design,
    nahw.study_disease,
    nahw.study_inc_dur,
    nahw.study_inc_temp,
    nahw.study_name,
    nahw.study_tmnt,
    nahw.subclone,
    nahw.subgroup,
    nahw.subject_is_affected,
    nahw.submitted_sample_id,
    nahw.submitted_subject_id,
    nahw.submitter_handle,
    nahw.subspecf_gen_lin,
    nahw.sub_species,
    nahw.subsrc_note,
    nahw.substrain,
    nahw.substrate,
    nahw.substructure_type,
    nahw.subtype,
    nahw.sulfate,
    nahw.sulfate_fw,
    nahw.sulfide,
    nahw.super_population_code,
    nahw.super_population_description,
    nahw.surface_orientation,
    nahw.surf_air_cont,
    nahw.surf_humidity,
    nahw.surf_material,
    nahw.surf_moisture,
    nahw.surf_moisture_ph,
    nahw.surf_temp,
    nahw.suspend_part_matter,
    nahw.suspend_solids,
    nahw.sym_life_cycle_type,
    nahw.tan,
    nahw.teleomorph,
    nahw.temp,
    nahw.tertiary_treatment,
    nahw.tidal_stage,
    nahw.tillage,
    nahw.time_last_toothbrush,
    nahw.timepoint,
    nahw.time_since_last_wash,
    nahw.tiss_cult_growth_med,
    nahw.tissue,
    nahw.tissue_lib,
    nahw.toluene,
    nahw.tot_carb,
    nahw.tot_depth_water_col,
    nahw.tot_diss_nitro,
    nahw.tot_inorg_nitro,
    nahw.tot_iron,
    nahw.tot_mass,
    nahw.tot_nitro,
    nahw.tot_nitro_cont_meth,
    nahw.tot_n_meth,
    nahw.tot_org_carb,
    nahw.tot_org_c_meth,
    nahw.tot_part_carb,
    nahw.tot_phosp,
    nahw.tot_phosphate,
    nahw.tot_sulfur,
    nahw.train_line,
    nahw.travel_out_six_month,
    nahw.treatment,
    nahw.trophic_level,
    nahw.turbidity,
    nahw.twin_sibling,
    nahw.type_of_symbiosis,
    nahw.type_status,
    nahw.type_strain,
    nahw.typ_occupant_dens,
    nahw.upstream_intervention,
    nahw.urine_collect_meth,
    nahw.urogenit_disord,
    nahw.urogenit_tract_disor,
    nahw.vaccine_received,
    nahw.variety,
    nahw.ventilation_rate,
    nahw.ventilation_type,
    nahw.vfa,
    nahw.vfa_fw,
    nahw.virus_enrich_appr,
    nahw.virus_isolate_of_prior_infection,
    nahw.volatile_org_comp,
    nahw.wastewater_type,
    nahw.water_content,
    nahw.water_content_soil,
    nahw.water_content_soil_meth,
    nahw.water_current,
    nahw.water_cut,
    nahw.watering_regm,
    nahw.water_ph,
    nahw.water_source_shared,
    nahw.water_temp_regm,
    nahw.weekday,
    nahw.weight_loss_3_month,
    nahw.wet_mass,
    nahw.wga_amp_appr,
    nahw.wind_direction,
    nahw.window_open_freq,
    nahw.wind_speed,
    nahw.ww_endog_control_1,
    nahw.ww_endog_control_1_conc,
    nahw.ww_endog_control_1_protocol,
    nahw.ww_endog_control_1_units,
    nahw.ww_endog_control_2,
    nahw.ww_endog_control_2_conc,
    nahw.ww_endog_control_2_protocol,
    nahw.ww_endog_control_2_units,
    nahw.ww_flow,
    nahw.ww_industrial_effluent_percent,
    nahw.ww_ph,
    nahw.ww_population,
    nahw.ww_population_source,
    nahw.ww_pre_treatment,
    nahw.ww_primary_sludge_retention_time,
    nahw.ww_processing_protocol,
    nahw.ww_sample_duration,
    nahw.ww_sample_matrix,
    nahw.ww_sample_salinity,
    nahw.ww_sample_site,
    nahw.ww_sample_type,
    nahw.ww_surv_jurisdiction,
    nahw.ww_surv_system_sample_id,
    nahw.ww_surv_target_1,
    nahw.ww_surv_target_1_conc,
    nahw.ww_surv_target_1_conc_unit,
    nahw.ww_surv_target_1_extract,
    nahw.ww_surv_target_1_extract_unit,
    nahw.ww_surv_target_1_gene,
    nahw.ww_surv_target_1_known_present,
    nahw.ww_surv_target_1_protocol,
    nahw.ww_surv_target_2,
    nahw.ww_surv_target_2_conc,
    nahw.ww_surv_target_2_conc_unit,
    nahw.ww_surv_target_2_extract,
    nahw.ww_surv_target_2_extract_unit,
    nahw.ww_surv_target_2_gene,
    nahw.ww_surv_target_2_known_present,
    nahw.ww_surv_target_2_protocol,
    nahw.ww_temperature,
    nahw.ww_total_suspended_solids
   FROM non_attribute_metadata nam
     JOIN ncbi_attributes_harmonized_wide nahw ON nam.raw_id = nahw.raw_id;


-- public.ncbi_attributes_all_long_with_accessions source

CREATE OR REPLACE VIEW public.ncbi_attributes_all_long_with_accessions
AS SELECT nam.accession,
    naal.raw_id,
    naal.attribute_name,
    naal.harmonized_name,
    naal.display_name,
    naal.unit,
    naal.value
   FROM ncbi_attributes_all_long naal
     JOIN non_attribute_metadata nam ON naal.raw_id = nam.raw_id;

Integrate more data, like a SQL representation of MIxS and the EnvO semantic SQL database

In psql, as an administrative user:

create database envo;

obtain or build the EnvO semantic-sql SQLlite file with something like

curl --output envo.db.gz https://s3.amazonaws.com/bbop-sqlite/envo.db.gz
gunzip envo.db.gz

back at the OS shell, load with connection strings like this:

pgloader sqlite://path/to/envo.db pgsql://postgres@localhost:15432/ncbi_biosamples_feb26

better capture and use of BioProject identifers

Right now, numeric BioProject identified, concatenated with "|||" are are available for some BioSamples in the bp_id column

It might be preferable to have a many:many BioSample id to BioProject accession lookup

Do we need to integrate any other information form BioProject or SRA?

improve chaining of `pre-basex-all`, `basex-all` and `postgres-all` targets

basex-all includes load-biosample-sets, but make determines $(BIOSAMPLE-SET-XML-CHUNK-NAMES) when the the make process is first started, at which point the directory is empty, because the Biosample XML file hasn't been downloaded, unpacked,or split yet.

Switch to a target/recipe that get the directory listing

BIOSAMPLE-SET-XML-CHUNK-FILES=$(shell ls shared-chunks)

BIOSAMPLE-SET-XML-CHUNK-NAMES=$(notdir $(basename $(BIOSAMPLE-SET-XML-CHUNK-FILES)))

biosample_set_from_%:
	echo $@
	date
	time docker exec -it biosample-basex basex -c "CREATE DB $@ basex/shared-chunks/$@.xml"

load-biosample-sets: $(BIOSAMPLE-SET-XML-CHUNK-NAMES) # 5 hours? could possibly do in parallel on a big machine

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.