https://delaramglp.github.io/vair/

{ 
  "p": "GRC20",
  "op": "mint",
  "tick": "GitHub",
  "amt": "2000"
}

Great knowledge base, Patrick and team! Trusted and responsible AI is one of our five MLSecOps Community pillars; would you be open to adding our community as a resource please? Welcome to use either the link to our website (https://mlsecops.com/) or Slack community invitation, whichever you see fit (https://join.slack.com/t/mlsecops/shared_invite/zt-24f8mmm45-Qc9qfZVxzBL4J5vcuLjaVw).

Well done, all.

I don't want to post/endorse this blog, but seems there are good references: https://www.linkedin.com/pulse/red-teaming-assurance-accountability-jiahao-chen-pzj9e/.

[1] AI Red-Teaming Is Not a One-Stop Solution to AI Harms
[2] How easy is it to make the AI behind chatbots go rogue? Hackers at Defcon test it out
[3] OpenAI Red Teaming Network; Red Teaming Language Models to Reduce Harms; Microsoft AI Red Team building future of safer AI
[4] Report and Analysis of Political Exercises, September 1958; The Political Exercise - A Progress Report (March 1961)
[5] Present & Future ORSA Trends - A Forecast for the US Army
[6] A Quest for Excellence
[7] Red Teaming: A Means to Military Transformation
[8] Program Management in Design and Development
[9] Computer Security Technology Planning Study
[10] Tiger teams; corporations can follow government's lead in establishing special computer security squads
[11] Red Teaming of Advanced Information Assurance Concepts
[12] Organization and Training of a Cyber Security Team
[13] Red Teaming: Past and Present
[14] Cybersecurity Attacks – Red Team Strategies
[15] Effectiveness of cybersecurity audit
[16] The IIA's Three Lines Model
[17] NIST Framework for Improving Critical Infrastructure Cybersecurity
[18] What are you doing to prevent cyberattacks?
[19] Red Teaming Language Models with Language Models
[20] Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
[21] Red-Teaming Large Language Models
[22] When ‘red-teaming’ AI isn’t enough
[23] Response to Politico's "When ‘red-teaming’ AI isn’t enough"

@datherton09 - can you look through this references list for any nice resources.

https://fraser.stlouisfed.org/files/docs/historical/frbdal/circulars/frbdallas_circ_19820603_no82-063.pdf

For official guidance

Re section: Comprehensive Software Examples and Tutorials (Interpreting Machine Learning Models with the iml Package)
I am getting the following error when trying to execute the following code in section entitled "Replication requirements" (https://uc-r.github.io/iml-pkg):

classification data

df <- rsample::attrition %>%
mutate_if(is.ordered, factor, ordered = FALSE) %>%
mutate(Attrition = recode(Attrition, "Yes" = "1", "No" = "0") %>% factor(levels = c("1", "0")))

Error: 'attrition' is not an exported object from 'namespace:rsample'

The problem was solved using the following code:
(...)
h2o.init()

data

library(modeldata)
data("attrition", package = "modeldata")

classification data

df <- attrition %>%
mutate_if(is.ordered, factor, ordered = FALSE) %>%
mutate(Attrition = recode(Attrition, "Yes" = "1", "No" = "0") %>% factor(levels = c("1", "0")))

Unfortunately, I got another error after trying to execute the following code (section entitled "Global interpretation/Feature importance" (https://uc-r.github.io/iml-pkg):

compute feature importance with specified loss metric

imp.glm <- FeatureImp$new(predictor.glm, loss = "mse")
imp.rf <- FeatureImp$new(predictor.rf, loss = "mse")
imp.gbm <- FeatureImp$new(predictor.gbm, loss = "mse")

Error in [.data.frame(prediction, , self$class, drop = FALSE) : undefined columns selected
Error in [.data.frame(prediction, , self$class, drop = FALSE) : undefined columns selected
Error in [.data.frame(prediction, , self$class, drop = FALSE) : undefined columns selected

I use R 4.2.0/ Win10

We should really update from master to main, but I don't know what happens in this case. Does anyone else?

super awesome list, have you considered adding github stars for each repo like was done here?:
https://github.com/wangyongjie-ntu/Awesome-explainable-AI

Makes it especially useful for folks who are looking for a well-maintained solution.

https://www.gov.uk/government/news/prime-minister-calls-for-global-responsibility-to-take-ai-risks-seriously-and-seize-its-opportunities
find the paper

https://www.homeaffairs.gov.au/foi/files/2023/fa-230400097-document-released-part-1.PDF
https://www.vera-2r.nl/development

https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
https://assets.iqt.org/pdfs/IQTLabs_RoBERTaAudit_Dec2022_final.pdf/web/viewer.html
https://docs.google.com/presentation/d/1m0p1KcYw-HM6U0hr_8egH2-7v4SlQhJq/edit
https://hackerone.com/twitter-algorithmic-bias?type=team
https://incidentdatabase.ai/
https://avidml.org/
https://github.com/nyu-mll/bbq
https://allenai.org/data/real-toxicity-prompts
https://www.promptingguide.ai
https://syntheticmedia.partnershiponai.org/
https://github.com/rudinger/winogender-schemas
https://github.com/sylinrl/TruthfulQA
https://arxiv.org/pdf/2005.00816.pdf
https://blogs.gwu.edu/law-eti/ai-litigation-database/

Hey! Thanks for compiling this great list of XAI work.

Consider adding Quantus to the list of evaluation packages (published in JMLR (2023), paper). Thanks.

VAIR - see other issue, NIST Vocab, etc.

• https://forhumanity.center/bok/? switchout with existing ForHumanity link? Also just feels like community guidance?
• https://www.theiia.org/globalassets/documents/content/articles/gpi/2017/december/gpi-artificial-intelligence-part-ii.pdf, seems more targeted
• https://www.auditboard.com/blog/ai-auditing-frameworks/ ?
• googling "ai audit checklist"
• more?

find privacy policies

See R section for example

• It would be nice to have a list of current contributors and update this list as more people add resources to this repo.

Something like awesome-responsible-machine-learning? Does anyone know what happens if we change the name?

• Accessibility and Inclusion

  • Adaptive Technologies
  • Design Principles
  • User Studies

• Algorithmic Bias

  • Sources of Bias
  • Mitigation Strategies
  • Impact Assessment

• BibTeX (Various)

• Cultural and Geographical Context

  • Localized Practices
  • Cross-Cultural Studies
  • Cultural Sensitivity

• Data Privacy

  • Anonymization Techniques
  • Legal Frameworks
  • Privacy vs Utility

• Environmental Impact

  • Energy Consumption
  • Sustainable Practices
  • Ecological Ethics

• Ethical ML

  • Foundational Theories
  • Case Studies
  • Ethical Guidelines

• Explainability and Interpretability

  • Technical Approaches
  • User Experience
  • Legal Requirements

• Feminist Approaches to ML

  • Gender Bias
  • Feminist Theory
  • Empowerment

• Historical Perspectives

  • Early Computational Models
  • Historical Context
  • Future Histories

• Human-Centered ML

  • User-Centric Design
  • Human-in-the-Loop
  • Cognitive Psychology

• Indigenous Perspectives

  • Data Sovereignty
  • Cultural Respect
  • Case Studies

• ML in Healthcare

  • Clinical Applications
  • Ethical Considerations
  • Regulatory Landscape

• Regulation and Policy

  • Current Legislation
  • Policy Proposals
  • Enforcement and Oversight

• Social Impact of ML

  • Positive Impacts
  • Negative Impacts
  • Societal Discussions

• Sociotechnical Systems

  • Integration Challenges
  • System Theory
  • Ethnographic Studies

• Web (Various)

Add FTC blog posts on AI

I tried to add this awesome on the main awesome list but it doesn't follow some guidelines:

Missing guidelines
"Main awesome"

Whats the author/contributors opinion on changing the format so this nice awesome can make the list?

https://a16z.com/2023/05/25/ai-canon/

Firstly, thanks! This page is awesome.

I really don't like to look a gift horse in the mouth, but the book "An Introduction to Machine Learning Interpretability" by Gill and Hall is not free. It's paywalled: you need a Safari account to access it, which isn't free.

Best wishes,

John

Pai synthetic media tool kit
Llama 2 responsible use guide
DAIR prompt guide

https://gitlab.jatic.net/home and frameworks link as sub bullet.

• Infographics such as: https://fpf.org/wp-content/uploads/2021/01/FPF_AIEcosystem_illo_03.pdf

In addition to FTC AI blogs, let's add:

• 2021 OCC Model Risk Management Handbook
• Finalized version of NIST Four principles of XAI
• Finalized FDA SaMD guidance
• FHA model risk management/model governance guidance
• FFEIC model risk or AI guidance

Note to Self: Guide's ethical scope is universal and multidisciplinary. To enrich, add resources on geo-cultural ethical variations. Aim for depth and breadth. Potential sections with viewpoints from subfields of ethics, sociology, law for a fuller responsible ML perspective?

• AI ID
• Avid
• aiaaic (?)
• the void
• GWU litigation database

Start here: https://medium.com/rti-cds/12-conferences-related-to-responsible-data-science-and-ai-1a076a22fc71 ?

• Tensor flow projector - “An application to visualized tensorflow work.”
• h2o - borrow from R descriptions
• Rudin group (Bayesian rule list, Bayesian And/Ors) - Implementation of X by the Rudin group at Duke.

https://uit.stanford.edu/security/responsibleai, if this isn't here, lets add it (despite it coming from the world's greatest techwashing institution ... )

When h2o-3 is mentioned, reference this GitHub: https://github.com/h2oai/h2o-3

https://www.coursera.org/learn/introduction-to-generative-ai?trk_ref=articleProductCard

Maybe some other courses here, let's only list if free:
https://www.coursera.org/articles/what-is-generative-ai

When referencing scikit-learn, use https://github.com/scikit-learn/scikit-learn.

People like software and it's kind what this list was known for, so:

* xplique
* ExplainaBoard

People are insane about GAI, so:
* Google Privacy & Terms, Generative AI Prohibited Use Policy

Other interests:
* Nvidia MLPerf
* NewsGuard AI Tracking Center

thoughts @datherton09 ?

https://creativecommons.org/licenses/

@lisahdd007

https://www.isaca.org/resources/isaca-journal/issues/2021/volume-3/information-system-contingency-planning-guidance

(Community guidance, I think )

Look for python packages, lists of lists etc using trustworthy AI desiderata.

Consider updating tags, including privacy, reliability, NIST, etc.

Build out the domain-specific software section.

Hi, any API or python package for adaptive incremental models that uses online batch stream of data? Thanks !!

Hi @jphall663. The following articles deal with the debugging prospects of Deep Neural Networks:

• Visualizing and debugging deep convolutional networks
• Troubleshooting Deep Neural Networks
• Things I wish we had known before we started our first Machine Learning project
• Practical Advice for Building Deep Neural Networks
• Top 6 errors novice machine learning engineers make
• A Recipe for Training Neural Networks
• Training Neural Nets: a Hacker’s Perspective

In which section can we place these?

• algorithmic accountability act or other potential data privacy or AI regulation
• get the versioning of the bill right

{ 
  "p": "GRC20",
  "op": "mint",
  "tick": "GitHub",
  "amt": "2000"
}

https://drive.google.com/drive/folders/1WQPaL-ozhZbZaDichFm4gWZQpGwriT32

Anything good in here for community guidance or other catergories?

Basic re-intro plan:

• Implement Danny's categories
  • Highlight places where we need the most help:
    • Communities and Forums
    • Conferences and Workshops
    • Challenges and Competitions
• Is there a way to alert previous contributors for the same purposes?
• Incorporate internal feedback
• Broader announce (Tentatively, Oct. 1)

• executive order on AI
• NIST 800-30 rev1 https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-30r1.pdf
• IEEE 1012 (199X or 2016) https://people.eecs.ku.edu/~hossein/Teaching/Stds/1012.pdf
• https://www.uspto.gov/sites/default/files/documents/USPTO_AI-Report_2020-10-07.pdf
• https://www.commerce.gov/issues/intellectual-property (see if you think it misses the mark, it might b/c I don't see an AI focus)
• https://standards.ieee.org/ieee/3119/10729/

https://www.energy.gov/ai/artificial-intelligence-technology-office

Both the NeurIPS workshop and ICML links were dead or went to general conference pages. I removed for now. Let's try to find the new or direct links. NeurIPS also runs an AI financial services workshop that might qualify .