Topic: data-profiling Goto Github

👇 Here are 69 public repositories matching this topic...

ahmadassaf / roomba
data-profiling,A Node.js tool to examine the correctness of Open Data Metadata and build custom dataset profiles
User: ahmadassaf
Home Page: http://roomba.ahmadassaf.com
node data-quality dataset portal ckan ckan-api dataset-catalog dataset-metadata data-profiling

apicrafter / metacrafter
data-profiling,Metadata and data identification tool and Python library. Identifies PII, common identifiers, language specific identifiers. Fully customizable and flexible rules
Organization: apicrafter
metadata pii pii-detection data-profiling entity-recognition datadiscovery

baligoyem / dataqtor
data-profiling,🔍Your Data Quality Detector / Gain insight into your data and get it ready for use before you start working with it 💡📊🛠💎
User: baligoyem
Home Page: https://share.streamlit.io/baligoyem/dataqtor/main/home.py
data-quality data-analysis data-quality-checks data-quality-measurement data-quality-assessment data-manipulation data-quality-monitoring streamlit pandas data-profiling

camillereaves / subreddit-crossposting
data-profiling,Map naturally-occurring inter-subreddit content sharing patterns on Reddit by analyzing how posts are “cross-posted" between subreddits based on 2.5 million posts across the top 2,500 subreddits. Uses ECL and HPCC Systems.
User: camillereaves
ecl hpcc reddit mapping-tools data-analysis data-analysis-in-ecl data-mining social-network-analysis hpcc-systems data-cleaning

christianbors / openrefinequalitymetrics
data-profiling,MetricDoc is an interactive visual exploration environment for assessing data quality
User: christianbors
data-quality-checks data-quality data-profiling data-wrangling quality-metrics visual-analytics interactive-visualizations

cleanlab / cleanlab
data-profiling,The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
Organization: cleanlab
Home Page: https://cleanlab.ai
weak-supervision data-cleaning data-quality data-science noisy-labels data-centric-ai out-of-distribution-detection outlier-detection active-learning data-labeling

cleanlab / cleanlab-studio
data-profiling,Client interface for all things Cleanlab Studio
Organization: cleanlab
Home Page: https://help.cleanlab.ai/
annotations automl computer-vision data-centric-ai data-labeling data-profiling data-quality data-science data-validation machine-learning

cleanlab / cleanvision
data-profiling,Automatically find issues in image datasets and practice data-centric computer vision.
Organization: cleanlab
Home Page: https://cleanvision.readthedocs.io/
computer-vision data-centric-ai data-exploration data-quality data-validation deep-learning exploratory-data-analysis image-analysis image-classification image-generation

codelionx / distod
data-profiling,DISTOD algorithm: Distributed discovery of bidirectional order dependencies
User: codelionx
akka akka-cluster bidirectional-order-dependencies data-profiling distributed-systems elastic java order-dependencies order-dependency-discovery scala

cods-gcs / kglids
data-profiling,Linked Data Science powered by Knowledge Graphs
Organization: cods-gcs
data-profiling datascience knowledge-graph linked-data-science pipelines

darenasc / auto-fes
data-profiling,Automated exploration of files in a folder structure to extract metadata and potential usage of information.
User: darenasc
data-exploration data-profiling data-science plain-text python

datakitchen / data-observability-installer
data-profiling,Installer for DataKitchen's Open Source Data Observability Products. Data breaks. Servers break. Your toolchain breaks. Ensure your team is the first to know and the first to solve with visibility across and down your data estate. Save time with simple, fast data quality test generation and execution. Trust your data, tools, and systems end to end.
Organization: datakitchen
Home Page: https://datakitchen.io
data data-engineering data-observability data-profiling data-quality data-science datachecker datacleaner datacleaning dataops dataquality datatesting datavalidation mssql pipeline-tests postgresql redshift self-hosted snowflake

desbordante / desbordante-core
data-profiling,Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.
Organization: desbordante
data-analytics data-cleaning data-cleansing data-engineering data-exploration data-mining data-profiling data-science data-wrangling data-preprocessing

dm4ml / gate
data-profiling,Drift detection module for machine learning pipelines.
Organization: dm4ml
Home Page: https://dm4ml.github.io/gate/
data-profiling data-science dataset-shifts drift drift-detection machine-learning mlops python

dqops / dqo
data-profiling,Data Quality and Observability platform for the whole data lifecycle, from profiling new data sources to full automation with Data Observability. Configure data quality checks from the UI or in YAML files, let DQOps run the data quality checks daily to detect data quality issues.
User: dqops
Home Page: https://dqops.com/docs/
data-ops data-quality data-quality-checks data-quality-measurement data-quality-monitoring data-quality-report monitoring data-observability data-profiling

fbdesignpro / sweetviz
data-profiling,Visualize and compare datasets, target values and associations, with one line of code.
User: fbdesignpro
pandas-dataframe eda pandas data-exploration data-analysis data-science data-visualization machine-learning data-profiling exploratory-data-analysis

gandalf1819 / nycopendata-profiling-analysis
data-profiling,Open Data Profiling, Quality and Analysis on NYC OpenData dataset with semantic profiling using fuzzy ratio, Levenshtein distance and regex
User: gandalf1819
data-profiling nyc-opendata nyc-311-dataset big-data pyspark hdfs modin dask dask-distributed pandas

giagiannis / data-profiler
data-profiling,Data profiler is an attempt to model the behavior of a given operator for a set of datasets.
User: giagiannis
bhattacharyya-coefficient data-modeling data-profiling data-science dataset machine-learning similarity-matrix

great-expectations / great_expectations
data-profiling,Always know what to expect from your data.
Organization: great-expectations
Home Page: https://docs.greatexpectations.io/
pipeline-tests dataquality datacleaning datacleaner data-science data-profiling pipeline pipeline-testing cleandata dataunittest

hi-primus / bumblebee
data-profiling,🚕 A spreadsheet-like data preparation web app that works over Optimus (Pandas, Dask, cuDF, Dask-cuDF, Spark and Vaex)
Organization: hi-primus
Home Page: https://hi-bumblebee.com/
data-profiling data-cleaning bumblebee gui data-preparation python dask optimus gpu cudf

hi-primus / optimus
data-profiling,:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Organization: hi-primus
Home Page: https://hi-optimus.com
spark pyspark data-wrangling bigdata big-data-cleaning data-science data-cleansing data-cleaner data-transformation machine-learning

hpcc-systems / datapatterns
data-profiling,HPCC Systems ECL bundle that provides some basic data profiling and research tools to an ECL programmer
Organization: hpcc-systems
data-profiling ecl-bundle hpcc-platform hpcc-systems

infuseai / piperider
data-profiling,Code review for data in dbt
Organization: infuseai
Home Page: https://www.piperider.io/
data-pipeline data-profiling data-quality data-science data-exploration eda exploratory-data-analysis data-testing python data-observability data-profiler data-reliability data-visualization dbt dbt-metrics code-review reporting pull-requests continuous-integration

ing-bank / popmon
data-profiling,Monitor the stability of a Pandas or Spark dataframe ⚙︎
Organization: ing-bank
Home Page: https://popmon.readthedocs.io/
population-monitoring monitoring data-science python data-distributions statistics data-profiling statistical-tests statistical-process-control pandas

ismaildawoodjee / greatex
data-profiling,A project for exploring how Great Expectations can be used to ensure data quality and validate batches within a data pipeline defined in Airflow.
User: ismaildawoodjee
docker airflow great-expectations python data-quality ecommerce csv sql postgres parquet

lieseb-1746743 / data-cleaning
data-profiling,Data cleaning tool.
User: lieseb-1746743
cleaning-data data-profiling data-cleaning data-cleansing data-clustering

mtna / rds-js
data-profiling,Simplify usage of the RDS API for TypeScript/JavaScript developers
Organization: mtna
Home Page: https://mtna.github.io/rds-js/
rich-data-services rds metadata-technology-north-america mtna metadata typescript javascript data-profiling data-mapping data-transformation

mtna / rds-js-examples
data-profiling,TypeScript/JavaScript example code using the RDS API
Organization: mtna
Home Page: https://mtna.github.io/rds-js-examples/
rich-data-services rds metadata-technology-north-america mtna metadata typescript javascript data-profiling data-mapping data-transformation

mtna / rds-r
data-profiling,R package to simplify the usage of the RDS REST API and provide convenience in accessing data and metadata.
Organization: mtna
Home Page: https://www.richdataservices.com
rich-data-services rds metadata-technology-north-america mtna metadata data-profiling data-mapping data-transformation data-validation data-ingestion

open-metadata / openmetadata
data-profiling,Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.
Organization: open-metadata
Home Page: https://open-metadata.org
metadata datadiscovery data-science dataquality data-profiling metadata-management dataengineering data-catalog data-observability dbt

open-metadata / openmetadata-site
data-profiling,Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.
Organization: open-metadata
Home Page: https://open-metadata.org
automation bigdata bigdataanalytics data-catalog data-discovery data-observability data-profiling data-quality-monitoring data-science datadiscovery

opendatadiscovery / odd-collector
data-profiling,Open-source metadata collector based on ODD Specification
Organization: opendatadiscovery
data-catalog data-discovery data-governance data-observability data-platform data-profiling datacatalog lineage data-piplines hacktoberfest

opendatadiscovery / odd-collector-gcp
data-profiling,Open-source GCP metadata collector based on ODD Specification
Organization: opendatadiscovery
data-catalog data-discovery data-platform lineage data-governance data-profiling datacatalog data-observability data-pipelines hacktoberfest

opendatadiscovery / odd-platform
data-profiling,First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
Organization: opendatadiscovery
Home Page: https://opendatadiscovery.org
oss data-platform metadata metadata-management data-pipelines data-engineering observability data-catalog datacatalog data-discovery

polyaxon / haupt
data-profiling,Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
Organization: polyaxon
tensorflow deep-learning jupyter python pytorch machine-learning models ui visualization matplotlib

polyaxon / traceml
data-profiling,Engine for ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.
Organization: polyaxon
pandas pandas-summary dataframes data-science spark dask plotly statistics matplotlib data-profiling data-visualization data-exploration dataops mlops data-quality data-quality-checks explainable-ai pytorch tensorflow tracking

psebenick / data-profiling
data-profiling,a set of scripts to pull meta data and data profiling metrics from relational database systems
User: psebenick
data-profiling database hive inventory metadata oracle python sql sqlserver

raymon-ai / raymon
data-profiling,The official http://raymon.ai data profiling and logging library.
Organization: raymon-ai
machine-learning artificial-intelligence data-profiling data-validation logging-library

rounayak / data-profiling-tool
data-profiling,The program compares two files at a time and does the following 1.Gathering metadata on the individual tables(column count,record count,list of columns with datatype etc) 2.Identifying matching columns between tables based on names as well as data. Using machine learning, we are handling syntactic as well as semantic variations of column names for accurate matching. 3. Finding duplicate columns in single table with the option to deduplicate if required 4. Finding columns with missing data/null values.
User: rounayak
python data-profiling

samapatil1729 / tools-in-data-science
data-profiling,This course will teach students to use popular tools for sourcing data, transforming it, building and optimizing models, communicating these as visual stories, and deploying them in production.
User: samapatil1729
clustering data-aggregation data-cleaning data-modeling data-profiling data-transformation scraping

sjtu-quant / awesome-ml-data-quality-papers
data-profiling,Papers about training data quality management for ML models.
Organization: sjtu-quant
ai4db data-debugging data-management data-profiling data-quality data-valuation db4ai machine-learning data-centric-ai

sodadata / soda-core
data-profiling,:zap: Data quality testing for the modern data stack (SQL, Spark, and Pandas) https://www.soda.io
Organization: sodadata
Home Page: https://go.soda.io/core-docs
python data-engineering data-governance data-monitoring data-observability data-profiling data-quality data-quality-checks data-quality-monitoring data-reliability

statsim / profile
data-profiling,Profile. Generate data profiles in the browser (work in progress)
Organization: statsim
Home Page: https://statsim.com/profile/
data-profiling data-profile statistics streaming-algorithms online-algorithms

swiple / swiple
data-profiling,Swiple enables you to easily observe, understand, validate and improve the quality of your data
Organization: swiple
Home Page: https://swiple.io
data data-observability data-quality observability python validation fastapi data-profiling data-science data-analytics

tsegall / fta
data-profiling,Metadata/data identification Java library. Identifies Semantic Type information (e.g. Gender, Age, Color, Country,...). Extensive country/language support. Extensible via user-defined plugins. Comprehensive Profiling support.
User: tsegall
semantic-type-detection semantic-typechecking semantic-types date java data-discovery data-profiling data-profiler metadata

vida-nyu / auctus
data-profiling,Dataset search engine, discovering data from a variety of sources, profiling it, and allowing advanced queries on the index
Organization: vida-nyu
crawling data-profiling dataset dataset-search index search search-engine

viperior / rimhistory
data-profiling,RimWorld game save data analyzer
User: viperior
data-engineering elt xml pandas data-visualization data-analysis data-science data-profiling simulation-data rimworld

wisemuffin / data-cicd-tpch
data-profiling,This repo shows how you can utilise CICD to speed up data development. Video link below.
User: wisemuffin
Home Page: https://www.youtube.com/watch?v=uotDffI33jg
dbt data-profiling data-quality data-cicd dbt-model

xxl4tomxu98 / data-engineering-python-great-expectations
data-profiling,Demo on Data Engineering using Great Expectations API
User: xxl4tomxu98
data-engineering data-pipeline great-expectations data-analytics data-mining data-profiling

ydataai / ydata-profiling
data-profiling,1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
Organization: ydataai
Home Page: https://docs.profiling.ydata.ai
big-data-analytics data-analysis data-exploration data-profiling data-quality data-science deep-learning eda exploration exploratory-data-analysis hacktoberfest html-report jupyter jupyter-notebook machine-learning pandas pandas-dataframe pandas-profiling python statistics