Code Monkey home page Code Monkey logo

data-docs's Introduction

data-docs

Public documentation about IMaRS data.

Documents in the ./docs directory here are in markdown format and provide information about how data is organized at IMaRS.

If you have a question about IMaRS data or want to make your data more available please open an issue in this repo's github issue tracker.

Special Access for IMaRS Researchers

IMaRS researchers can access data in the \\yin\lab_share\datashare\ folder from windows or /srv/imars-objects/homes/datashare on linux. These directories are managed with puppet by profile::imars_datashare_homelinks.

A basic overview of the directories there:

  • regions : datasets bounded by a "region". Regions are defined on a per-project basis. For more information consult the imars_product_metadata database managed with puppet by role::product_metadata_rows.sql
  • mirrored : datasets copied directly from another source
  • legacy : older datasets not comforming to current organizational standards
  • ingests : staging locations for incoming data
  • custom : other datasets not fitting any of the above

data-docs's People

Contributors

7yl4r avatar sebastiandig avatar

Watchers

 avatar  avatar  avatar  avatar

data-docs's Issues

ingest WV from Matt

I've gone through and delete about as much as I can for now. However, I also found a number of WV images that were directly ftp'd to me from PGC and do not appear to have made it into the extra_data directories. Are you able to transfer them into their respective date-based folders? They are located here:

X:\mmccarthy\Matt\USF\Other\CHIMMP\Circe\Raw\Images

Ingesting these is important for organizational purposes, but also to ensure we have as many images as possible.

create data flow for data

  • create .md file here
  • add row to spreadsheet for Matt (TODO: link)
  • register into bio eco portal
  • upload raw data to (where?)
  • upload to (datatype|domain) specific locations:
    • CTD : GCOOS, ERDDAP, NCEI
    • Bio optical : SEABASS
      • BB3
      • GER
    • taxa occurrences : OBIS
    • animal telemetry: ATN
  • update BIO ECO info with links to all the above
  • present data to MBON group

naming conventions between IMaRS & AOML need to be standardized

Naming conventions from AOML changed recently.

Sebastian reports that last summer's file naming pattern was different from 2023 spring.

Sebastian should email Marion (or someone else?).

two depth timestamp columns: one timestamp and one by second(?)

search for HPLC samples

search is ongoing. places searched:

  • makenzie spot 1 - found eDNA stuff instead
  • makenzie spot 2 - found eDNA stuff instead
  • ?

box.com rclone to servers

@sebastiandig has organized MBON cruise data in box.com. We can use rclone to download a copy of the data to IMaRS servers.


advice from Doug:

There is no recommended way since Box doesn’t officially support linux and most of USF IT works in Windows, not linux. I would say whatever way works best for you. There are people using webdav and I also had made some notes on using ftps to transfer files. I’ll send those to you directly in case you want to play with that method.


ftp.box.com notes are in my email.

CDOM data and Jenn

We are up to December 2022 as of today.

Starting with CDOM then will go back and do the pads.

CDOM processing takes 3 days.

Every week 4 days of filter pads.

12-24 samples/day.

Not doing duplicates but they are being separated. Doing the odds, saving the evens.
Jenn expressed that the duplicates looked good and processing both was not necessary.
SeaBASS data rarely has duplicates on filter pad data.

Reflectance Data + Mostafa's Dissertation

FMK wants Mostafa to work with the reflectance data.
David English is the lead on that analysis.
He has some code to help with this.
There is an IOCCG document on this.
Dan will send IOCCG document as a starting point, then Mostafa can connect with David directly.

research questions:

  1. find stations with multiple datasets collected at the same time to create time series. Molasses Reef, Western Sambo, Looe Key,
  2. characterize variation in plankton productivity in FL Keys & how it relates to ocean color. Need to look at biomass, optics, & satellite data to see changes in FL Keys.

Creating a vertical profile at each of the major stations for nutrients, chlorophyll, and whatever else you can would be a good start.
The can move towards comparison of surface productivity and reflectance.

HPLC submitted

Almost ready.

Sebastian is checking details of samples.
Verifying the sample IDs.

Documentation of Resources

Hi @7yl4r

I'm not sure where to put resources for optics information/processing. It would probably just be a folder with different .md files for parameters. Some theory, videos, doucments, code, etc.

Let me know where I can start putting information, whether in dat-docs or another repo all together.

Best,
@sebastiandig

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.