Everything Apache NiFi
Timothy Spann @PaaSDev
Updates
- https://github.com/tspannhw/FLiPN-NFT
- Apache NiFi 1.20.0 is now out. Check it out. https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
- https://www.youtube.com/watch?app=desktop&v=h503MibiENY&feature=youtu.be
- https://bryanbende.com/development/2021/07/19/apache-nifi-1-14-0-secure-by-default
- Latest Version - Apache NiFi 1.13.2! Don't use Apache NiFi 1.13.1
- Apache NiFi 1.13.2.2 - https://www.datainmotion.dev/2021/04/nifi-on-cloudera-data-platform-upgrade.html
- Apache NiFi 1.13.0 - https://www.datainmotion.dev/2021/02/new-features-of-apache-nifi-1130.html
- Apache NiFi 1.12 - https://twitter.com/pvillard31/status/1296469452180119553
- Apache NiFi on K8 - https://twitter.com/AppInt4All/status/1306643414725750784?s=20
- Questions on NiFi Jam - https://blog.cloudera.com/top-5-questions-about-apache-nifi/
Must Watch Video
- Apache NiFi Part 5: Primary Node / Complex Expression Language: https://www.youtube.com/watch?v=JbUjYr6Kd3I
- Apache NiFi Part 4: Scheduling: https://www.youtube.com/watch?v=pZq0EbfDBy4
- Apache NiFi Part 3: Load Balancing: https://www.youtube.com/watch?v=by9P0Zi8Dk8
- Apache NiFi Anti-Patterns Part 2: Flow Layout by Mark: https://www.youtube.com/watch?v=v1CoQk730qs&feature=youtu.be
- Apache NiFi Anti-Patterns, Part 1: by Mark: https://www.youtube.com/watch?v=RjWstt7nRVY
Install
- https://www.cloudera.com/downloads/cdf/cfm-trial.html
- http://nifi.apache.org/download.html
- https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
Don't Run NiFi Batch Jobs... But you can. Try NiFi Stateless, but you don't want to use Airflow try this
- https://benyaakobi.medium.com/nifi-scheduling-a522a1c9e740
- http://funnifi.blogspot.com/2018/08/one-and-done-flows-aka-job-processing.html
- https://getindata.com/blog/nifi-ingestion/fast-and-easy-what-could-possibly-go-wrong-one-year-history-certain-nifi-flow/
- https://boristyukin.com/how-to-run-sqoop-from-nifi/
- https://www.hadoopinrealworld.com/building-a-data-pipeline-with-apache-nifi/
Support
- https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.2/release-notes/content/supported-nifi-processors.html
- https://docs.cloudera.com/management-console/cloud/connection-to-private-subnets/topics/mc-endpoint_access_gateway.html
Cloudera Data Flow - Kubernetes Deployment
DevOps / Tools / Scripts
- https://github.com/asdaraujo/data-in-motion
- https://github.com/asdaraujo/flink-kafka-replicator
- https://github.com/asdaraujo/kafka-client-examples
Load Balancing
- https://medium.com/@nlabadie/apache-nifi-syslog-load-balancing-c43d87480deb
- https://t.co/4ANiUPh706?amp=1
- https://community.cloudera.com/t5/Community-Articles/NiFi-Cluster-and-Load-Balancer/ta-p/248227
- https://community.cloudera.com/t5/Support-Questions/NIFI-S2S-communication-issue-between-two-secured-NIFI/td-p/193227
Syslog
- https://medium.com/@nlabadie/apache-nifi-netflow-to-syslog-117d46867ae1
- https://medium.com/@nlabadie/apache-nifi-sftp-csv-to-syslog-json-d9da6938defa
- https://medium.com/@nlabadie/apache-nifi-pulling-from-mysql-and-sending-to-syslog-181dd4ae969c
- https://nathanlabadie.com/apache-nifi-netflow-to-syslog/
- https://blog.davidvassallo.me/2019/04/09/consuming-netflow-using-nifi/
Grok
- http://grokconstructor.appspot.com/do/match#result
- https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.grok.GrokReader/additionalDetails.html
- https://stackoverflow.com/questions/59291548/how-to-use-nifi-extractgrok-properly
Documentation
- https://nifi.apache.org/docs.html
- https://nifi.apache.org/docs/nifi-docs/html/overview.html
- https://nifi.apache.org/docs/nifi-docs/html/getting-started.html
- https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
- https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html
- https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html
- https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
- https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
- https://nifi.apache.org/docs/nifi-docs/html/user-guide.html
- https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html
- https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
Demos / Examples
- https://www.cloudera.com/content/dam/www/marketing/resources/demos/demo-real-time-data-streaming-using-apache-kafka.landing.html
- https://developer.ibm.com/solutions/security/articles/distributed-real-time-threat-detection-engine/
- https://michalklempa.com/2016/10/using-nifi-to-ingest-snmp-into-avro/
- https://github.com/geoHeil/streaming-reference/tree/master
- https://blog.min.io/minio-events-with-apache-nifi/
- https://gist.github.com/neallee2012/3eb625094a46f8e3f0cc9028e04a5093
- https://medium.com/digital-transformation-and-platform-engineering/streaming-data-lake-and-analytics-with-apache-nifi-c651700b56db
Cybersecurity
Reprise Demos
Blogs
- http://lonnifi.blogspot.com/
- https://www.datainmotion.dev/
- https://blogs.apache.org/nifi/
- https://bryanbende.com/
- http://funnifi.blogspot.com/
- http://ijokarumawak.github.io/blog/
- https://pierrevillard.com/
- https://www.nifi.rocks/
- https://blog.cloudera.com/?s=nifi
- https://dzone.com/users/297029/bunkertor.html
- https://medium.com/@abdelkrim.hadjidj
- https://www.calculatedsystems.com/content
- https://mikethomsen.github.io/
- https://community.cloudera.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=false&q=nifi*
- https://mvnrepository.com/artifact/org.apache.nifi
- https://github.com/apache/nifi
- https://apachenifi.slack.com/join/shared_invite
- http://flink.apache.org/
- https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/
- https://www.youtube.com/watch?v=pZq0EbfDBy4
- https://medium.com/apache-nifi-security
- https://lonnifi.blogspot.com/
- https://community.cloudera.com/t5/Community-Articles/How-to-use-NiFi-to-write-API-data-to-CDP-CDW/ta-p/306770
- https://www.youtube.com/channel/UCBHGHVGIihzmuOM86D8IfHQ/featured
- http://lonnifi.blogspot.com/2019/11/back-pressure-prediction-deep-dive.html
- https://www.datacrafts.fr/NiFi-Democratizing-Record-Processors/
- https://www.datacrafts.fr/Error-Handling/
- https://github.com/hau-mal/articles/blob/master/nifi-ingest-weather-data-en.md
- https://medium.com/@ben2460/using-nifi-to-react-to-dns-changes-f1d3a348a595
- https://docs.cloudera.com/cdf-datahub/7.2.7/nifi-snowflake/topics/cfm-snowflake-egress.html
- https://medium.com/@kar9475/parameter-context-retry-flowfile-back-pressure-prediction-in-nifi-1-10-release-6c4aa7def082
- https://bryanbende.com/development/2021/02/17/apache-nifi-saml-keycloak
- https://community.cloudera.com/t5/Community-Articles/How-to-import-a-flow-to-NiFi-registry-in-CDP-Cloud/ta-p/308335
- http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/
- https://pierrevillard.com/2017/02/07/using-counters-in-apache-nifi/
- https://pierrevillard.com/2018/08/29/monitoring-driven-development-with-nifi-1-7/
- https://www.baeldung.com/iot-data-pipeline-mqtt-nifi
- https://www.clearpeaks.com/streaming-data-to-hive-using-nifi-and-kafka-in-a-kerberized-cdh-cluster/?utm_source=social&utm_medium=linkedin&utm_campaign=cdh-nifi-kafka-kerberos-blog-post
- https://github.com/medined/nifi-prototype
- https://boristyukin.com/building-near-real-time-big-data-lake-part-2/
- https://gpdb.docs.pivotal.io/connectors/apache-nifi/1-0/overview.html
- https://lonnifi.blogspot.com/2019/11/back-pressure-prediction-deep-dive.html
- https://community.cloudera.com/t5/Community-Articles/Using-NiFi-for-CDE-Jobs-Pipeline/ta-p/310122
- https://www.clearpeaks.com/connecting-nifi-with-activemq/
- https://medium.com/@Sushil_Kumar/data-quality-with-apache-nifi-9fa5223bbf54
FHIR / HL7 / Health / Medical
Containers / Docker / K8
- https://hub.docker.com/r/apache/nifi-registry
- https://hub.docker.com/r/apache/nifi/
- https://hub.docker.com/r/apache/nifi-toolkit
- https://hub.docker.com/r/apache/nifi-stateless
Examples
NiFi Frameworks
Sequencing and Ordering
Expression Language:
-
random Description: Returns a random whole number ( 0 to 2^63 - 1) using an insecure random number generator.
-
nextInt Description: Returns a one-up value (starting at 0) and increasing over the lifetime of the running instance of NiFi
-
${now():toNumber()}
-
SELECT RPATH(message, '/c_ip') as xxx from flowfile
get recordpath for things like nested structured in json or xml
Other Options
- https://gist.github.com/ijokarumawak/7e6158460cfcb0b5911acefbb455edf0
- https://medium.com/google-cloud/gcp-pub-sub-ordering-and-apache-nifi-5e1ac9a14752
- http://funnifi.blogspot.com/2018/08/database-sequence-lookup-with.html
- https://community.cloudera.com/t5/Support-Questions/Generate-Sequence-number-in-Apache-NIFI/td-p/220571
- http://www.randomnumberapi.com/api/v1.0/random
Ingest and Process Logs
- https://bryanbende.com/development/2017/10/04/apache-nifi-processing-multiline-logs
- https://www.tigeranalytics.com/blog/real-time-nifi-alerts-automated-log-capturing-spark/
- https://nathanlabadie.com/nifi-pulling-data-from-splunk/
- https://www.youtube.com/watch?v=4yBc7hHvaQU
- https://www.exabeam.com/siem/apache-nifi-ingest-syslog-data/
- https://dzone.com/articles/reading-sudo-logs-with-apache-nifi
- https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html
- https://blog.davidvassallo.me/2018/09/19/apache-nifi-from-syslog-to-elasticsearch/
- https://www.keedio.org/real-time-data-ingestion-processing-and-storage-of-syslog-using-apache-nifi-flink-kafka-and-cassandra/
- https://www.hadoopinrealworld.com/building-a-data-pipeline-with-apache-nifi/
- https://www.futurespace.es/en/apache-nifi-flujo-de-extraccion-validacion-transformacion-y-carga-de-ficheros-caso-de-uso-real/
- https://nxlog.co/documentation/nxlog-user-guide/apache-nifi.html
- http://bigapps.fr/apache-nifi-for-dataflow-and-real-time-streaming-with-apache-kafka/
Community Q&A
- https://community.cloudera.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=false&q=nifi
- https://apachenifi.slack.com/join/shared_invite
- https://community.cloudera.com/t5/Support-Questions/Nifi-CICD-with-mutiple-flows-having-controller-services/m-p/299771#M219835
- https://community.cloudera.com/t5/Support-Questions/How-to-add-custom-attributes-to-AttributesToJSON/td-p/308326
- https://community.cloudera.com/t5/Community-Articles/Using-NiFi-for-CDE-Jobs-Pipeline/ta-p/310122
- https://community.cloudera.com/t5/Support-Questions/NiFi-Load-balancing-internal-or-external/td-p/310131
- https://community.cloudera.com/t5/Support-Questions/Automatic-deployment-of-NiFi-Flow/m-p/311319#M224586
- https://community.cloudera.com/t5/Community-Articles/Change-NiFi-Flow-Using-Rest-API-Part-1/ta-p/244631
- https://community.cloudera.com/t5/Community-Articles/Update-NiFi-Connection-Destination-via-REST-API/ta-p/244211
- https://community.cloudera.com/t5/Community-Articles/Simple-NiFi-template-for-flow-management-datahub/ta-p/311733
- https://community.cloudera.com/t5/Community-Articles/Connecting-Nifi-to-Impala-using-Simba-JDBC-driver/tac-p/312714#M6179
- https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418
- https://stackoverflow.com/questions/66259273/nifi-kerberos-ticket-login-not-supported-by-this-nifi
- https://community.cloudera.com/t5/Support-Questions/PrometheusReportingTask-with-same-port-and-multiple-nifi/m-p/316092#M226712
- https://github.com/chadgeary/nifi
- https://nathanlabadie.com/recombining-multiline-logs-with/
- https://nathanlabadie.com/video-tutorial-nifi-netflow-and-csv-ingest/
- https://nathanlabadie.com/nifi-pulling-data-from-splunk/
- https://nathanlabadie.com/running-nifi-in-a-docker-container/
- https://nathanlabadie.com/adding-a-timestamp-to-elasticsearch-from-apache-nifi/
- https://nathanlabadie.com/video-tutorial/
- https://nathanlabadie.com/the-basics-how/
- https://nathanlabadie.com/streaming-from-elastic-to-syslog-via-apache-nifi/
- https://blog.octo.com/en/accelerating-nifi-flows-delivery-part-1/
- https://www.fullstackanalyst.io/blog/open-source/processing-adobe-analytics-data-feeds-with-apache-nifi-for-adobe-experience-platform/
SAAS Interfaces
Software Artefacts
- https://mvnrepository.com/artifact/org.apache.nifi
- https://github.com/apache/nifi
- https://hub.docker.com/r/apache/nifi/
Must Read Articles
- https://pierrevillard.com/best-of-nifi/
- http://lonnifi.blogspot.com/2020/01/rules-integration-with-apache-nifi.html
- https://www.calculatedsystems.com/post/what-is-nifi-the-core-concepts
- https://martin.atlassian.net/wiki/spaces/lestermartin/pages/740196355/NiFi+Cheat+Sheet
Conference Talks
- https://github.com/alopresto/slides
- http://slideshare.com/bunkertor
- https://github.com/tspannhw/dws2017sydney
- https://community.cloudera.com/t5/Community-Articles/Real-Time-Ingesting-and-Transforming-Sensor-and-Social-Data/ta-p/247202
- http://iotfusion.net/session/enterprise-iiot-edge-processing-with-apache-nifi-minifi-and-deep-learning/
- https://www.slideshare.net/bunkertor/deep-learning-on-hdp-2018-prague
- https://www.slideshare.net/bunkertor/minifi-and-apache-nifi-iot-in-berlin-germany-2018
- https://dataworkssummit.com/session/apache-deep-learning-101/
- https://github.com/tspannhw/StrataNYC2018
- https://feathercast.apache.org/2018/10/02/apache-deep-learning-101-timothy-spann
- https://www.slideshare.net/bunkertor/cdf-edge-to-ai-with-cem-april-9-2019-princeton-fod-meetup
- https://www.slideshare.net/bunkertor/the-edge-to-ai-deep-dive-barcelona-meetup-march-2019
- https://www.datainmotion.dev/2019/03/barcelona-dataworks-summit-march-2019.html
- https://dzone.com/articles/dataworks-summit-and-nosql-day-review-2019-washing
- https://www.datainmotion.dev/2019/05/dataworks-summit-dc-2019-report.html
- https://github.com/tspannhw/StrataNYC2019
- https://www.slideshare.net/Hadoop_Summit/eventdriven-messaging-and-actions-using-apache-flink-and-apache-nifi
- https://www.slideshare.net/BryanBende/apache-nifi-sdlc-improvements
Quick Tips
- https://community.cloudera.com/t5/Community-Articles/Connect-Nifi-to-Cloudera-Hive-with-Kerberos-and-TLS/ta-p/249476
- https://community.cloudera.com/t5/Community-Articles/Part-2-Nifi-flow-creation-to-parse-new-images-and-run-the/ta-p/244380
- https://docs.cloudera.com/cem/1.2.2/collecting-audit-events/topics/cem-setting-up-ConsumeWindowsEventLog.html
- https://community.cloudera.com/t5/Support-Questions/Nifi-Multiple-predicate-in-recordpath-filter/m-p/306604
- https://community.cloudera.com/t5/Support-Questions/Can-executesql-processor-store-state-for-incremental-loading/td-p/224975
- https://community.cloudera.com/t5/Community-Articles/Update-the-Contents-of-FlowFile-by-using-UpdateRecord/ta-p/248267
- https://www.nifi.rocks/record-path-cheat-sheet/
- https://www.nifi.rocks/expression-language-cheat-sheet/
- https://dev.to/tspannhw/one-minute-nifi-tip-calcite-sql-notes-561
- https://dev.to/tspannhw/new-and-improved-it-s-nifi-1k6o
- https://dev.to/tspannhw/quick-tip-nifi-json-cleanup-3o22
- https://dev.to/tspannhw/openssl-ssl-hosting-in-nifi-2k5f
- http://ijokarumawak.github.io/nifi/2017/01/27/nifi-s2s-local-to-aws/
- https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/building-a-dataflow/content/site-to-site.html
- https://www.calculatedsystems.com/post/the-nifi-expression-language
- https://martin.atlassian.net/wiki/spaces/lestermartin/pages/740196355/NiFi+Cheat+Sheet
- http://lonnifi.blogspot.com/2020/01/rules-integration-with-apache-nifi.html
- https://www.calculatedsystems.com/post/getting-started-with-apache-nifi-5-common-questions
- https://levelup.gitconnected.com/an-overview-of-apache-nifi-and-toolkit-cli-deployments-785978dbce3b
- Set to Integer SQL Type https://community.cloudera.com/t5/Support-Questions/Nifi-ExectueSQL-how-to-force-a-parameter-to-be-a-string/td-p/240117
- Make sure to use Cache Server https://community.cloudera.com/t5/Support-Questions/Could-not-connect-to-Distributed-Map-Cache-server-to-get/td-p/269340
- Static End-Point Security https://community.cloudera.com/t5/Support-Questions/Advanced-Update-Attribute-won-t-display-correctly-in-NiFi/td-p/306729
- Use -Record https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apache.nifi.processors.standard.QueryDatabaseTableRecord/index.html
- Merge Attributes https://community.cloudera.com/t5/Support-Questions/Nifi-Processors-status-merge-attributes-into-single-flowfile/td-p/308470
- https://community.cloudera.com/t5/Support-Questions/NIFI-quot-No-available-buckets-quot-for-saving-flow-version/td-p/308171
- https://medium.com/analytics-vidhya/managing-your-data-flows-with-apache-nifi-ef559be2fcf4
- https://springml.com/blog/apache-nifi-integration-on-google-cloud/
- https://ryan-templeton.medium.com/snowflake-resource-monitor-reports-delivered-to-slack-using-apache-nifi-8dfd4fc4d579
- https://pierrevillard.com/2016/03/13/get-data-from-dropbox-using-apache-nifi/
Understanding NiFi Internals / Architecture
- https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418
- https://community.cloudera.com/t5/Community-Articles/Understanding-NiFi-max-thread-pools-and-processor-concurrent/ta-p/248920
- https://community.cloudera.com/t5/Community-Articles/Leveraging-group-based-Ranger-authorization-in-NiFi/ta-p/248000
- https://community.cloudera.com/t5/Community-Articles/NiFi-Understanding-how-to-use-Process-Groups-and-Remote/ta-p/245486
- https://community.cloudera.com/t5/Community-Articles/How-to-address-JVM-OutOfMemory-errors-in-NiFi/ta-p/244431
- https://community.cloudera.com/t5/Community-Articles/Understanding-NiFi-processor-s-quot-Run-Duration-quot/ta-p/248921
- https://community.cloudera.com/t5/Community-Articles/Dissecting-the-NiFi-quot-connection-quot-Heap-usage-and/ta-p/248166
- https://community.cloudera.com/t5/Community-Articles/How-to-expand-existing-NiFi-cluster-fault-tolerance-using/ta-p/245082
- https://community.cloudera.com/t5/Community-Articles/How-to-determine-which-FlowFiles-are-associated-to-the-same/ta-p/249185
- https://community.cloudera.com/t5/Community-Articles/How-to-achieve-better-load-balancing-using-NiFi-s-Site-To/ta-p/246279
Cloud
- https://community.cloudera.com/t5/Community-Articles/Spark-Structured-Streaming-example-with-CDE/ta-p/302050
- https://community.cloudera.com/t5/Community-Articles/How-to-configure-NiFi-S2S-from-local-machine-to-CDP-Public/ta-p/301235
- https://community.cloudera.com/t5/Support-Questions/CDP-Public-Cloud-Data-Replication/m-p/309362
- https://community.cloudera.com/t5/Community-Articles/How-to-connect-Nifi-to-Kafka-in-Data-Hub-on-CDP/ta-p/294441
- https://community.cloudera.com/t5/Support-Questions/Can-GetHDFS-processor-catch-changes/td-p/201212
- https://community.cloudera.com/t5/Community-Articles/Simple-Change-Data-Capture-CDC-with-SQL-Selects-via-Apache/ta-p/308376
- https://community.cloudera.com/t5/Community-Articles/How-to-use-Sqoop-to-Hydrate-Cloudera-Data-Warehouse/ta-p/303037
- https://community.cloudera.com/t5/Community-Articles/Harnessing-the-Data-Lifecycle-for-Customer-Experience/ta-p/294910
- https://community.cloudera.com/t5/Community-Articles/How-to-use-NiFi-to-write-API-data-to-CDP-CDW/ta-p/306770
- https://community.cloudera.com/t5/Community-Articles/Harnessing-the-hybrid-Cloud-use-CDP-Cloud-to-evaluate-COVID/ta-p/305220
- https://community.cloudera.com/t5/Community-Articles/How-to-write-data-from-NiFi-to-Kudu-in-CDP-Public-Cloud/ta-p/300402
- https://cloudacademy.com/blog/moving-data-to-s3-with-apache-nifi/
- https://community.cloudera.com/t5/Community-Articles/Simple-NiFi-template-for-flow-management-datahub/ta-p/311733
- https://www.cloudera.com/tutorials/cdp-importing-rdbms-data-into-hive.html
- https://community.cloudera.com/t5/Community-Articles/Working-with-S3-Compatible-Data-Stores-via-Apache-NiFi/ta-p/244584
- https://research.nccgroup.com/2021/01/04/using-aws-and-azure-for-cost-effective-log-ingestion-with-data-processing-pipelines-for-siems/
ETL / ELT / CDC / Load / Ingest
- https://datahovel.com/2019/11/14/big-data-and-stream-processing-101-part-4-how-to-do-a-simple-key-value-enrichment-in-apache-nifi/
- https://thirdeyedata.io/etl-pipeline-demonstration-using-apache-nifi/
Dockerization / K8
Reporting Tasks
- https://www.nifi.rocks/developing-a-custom-apache-nifi-reporting-task/
- https://dev.to/tspannhw/sql-reporting-task-for-cloudera-flow-management-hdf-apache-nifi-260f
Error Handling
Templates and Examples
Sizing and Performance
- https://www.datainmotion.dev/2020/07/sizing-your-apache-nifi-cluster-for.html
- https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/
Reporting and Monitoring
Use Cases and Examples
- Retail Price Comparisons https://github.com/tspannhw/retail-dynamic-shelf-pricing/tree/main
- Sensors https://github.com/tspannhw/meetup-sensors
- Twitter to Cloud https://github.com/tspannhw/ClouderaNow2020
- Airline Sentiment https://github.com/tspannhw/airline-sentiment-streaming
- NiFi + Kafka Connect Pipeline HDFS https://github.com/tspannhw/NiFItoKafkaConnect
- Send Messages to Discord Channels https://github.com/tspannhw/NiFiToDiscord
- Capture Data to SOLR Search Indexes https://github.com/tspannhw/nifi-solr-example
- Post to Slack https://github.com/tspannhw/nifi-1.10-templates
- Hosting a Website in NiFi https://github.com/tspannhw/webdataingest
- Ingest/Export to Cloud Data Warehouse https://github.com/tspannhw/clouddatawarehouse
- Several use cases https://github.com/tspannhw/ApacheNiFiByExamples
- Hortonworks Gallery https://github.com/hortonworks-gallery/nifi-templates
- Apache Examples https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
- Cryptocurrency https://github.com/tspannhw/nifi-flows
- Energy Monitoring https://github.com/tspannhw/nifi-energy-monitoring
- Leprechaun Detector https://github.com/tspannhw/leprechaun-detector
- More Examples https://www.batchiq.com/nifi-flow-examples.html
- Email Extraction https://github.com/MikeThomsen/nifi-email-extraction-bundle
- Data Generation https://github.com/MikeThomsen/nifi-datageneration-bundle
- SQL Server and Kerberos https://www.exabeam.com/information-security/apache-nifi-sql-server-and-kerberos-authentication/
- Files and Netflow https://www.exabeam.com/siem/apache-nifi-ingest-syslog-data/
- MySQL to SYSLOG https://www.exabeam.com/siem/apache-nifi-send-from-mysql-to-syslog/
- https://yauaa.basjes.nl/UDF-ApacheNifi.html
Sources & Sinks
- HBase/Phoenix https://www.datainmotion.dev/2020/07/phoenix-hbase-nifi-resources.html
- Operational Database https://www.cloudera.com/tutorials/cdp-cod-phoenix-sql-using-hbase-nifi-kafka.html
- REST https://www.datainmotion.dev/2020/07/ingesting-all-weather-data-with-apache.html
- REST https://dev.to/tspannhw/cloudera-flow-management-101-let-s-build-a-simple-rest-ingest-to-cloud-datawarehouse-with-lowcode-powered-by-apache-nifi-2275
- Search https://dev.to/tspannhw/building-search-indexes-with-apache-nifi-streams-2ea7
- Twitter https://dev.to/tspannhw/harnessing-the-data-lifecycle-for-customer-experience-optimization-streaming-classifications-on-twitter-streams-2p0f
- Impala, Kudu, Kafka https://dev.to/tspannhw/read-apache-impala-apache-kudu-tables-and-send-to-apache-kafka-in-bulk-easily-with-apache-nifi-4n3g
- Google Analytics API https://dev.to/tspannhw/quicktip-ingesting-google-analytics-api-with-apache-nifi-mg1
- Image Upload / Slack https://dev.to/tspannhw/nifi-1-10-postslack-easy-image-upload-22mh
- Hive / ORC https://dev.to/tspannhw/using-grovepi-with-raspberry-pi-and-minifi-agents-for-data-ingest-to-parquet-kudu-orc-kafka-hive-and-impala-2h9k
- REST / Monitoring https://dev.to/tspannhw/monitoring-cloudera-edge-flow-manager-and-cloudera-streams-messaging-manager-with-apache-nifi-387k
- Generate JSON https://dev.to/tspannhw/generating-json-data-in-apache-nifi-4apo
- Kafka, REST for Stocks https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221
- MongoDB https://community.cloudera.com/t5/Community-Articles/Exporting-and-Importing-Data-from-MongoDB-in-the-Cloud-with/ta-p/249470
- PowerPoint https://community.cloudera.com/t5/Community-Articles/Converting-PowerPoint-Presentations-into-French-from-English/ta-p/248974
- Documents to HDFS https://community.cloudera.com/t5/Community-Articles/Scanning-Documents-into-Data-Lakes-via-Tesseract-MQTT-Python/ta-p/248492
- Flight Data https://community.cloudera.com/t5/Community-Articles/Ingesting-Flight-Data-ADS-B-USB-Receiver-with-Apache-NiFi-1/ta-p/247940
- Street Cameras https://community.cloudera.com/t5/Community-Articles/Ingesting-and-Analyzing-Street-Camera-Data-from-Major-US/ta-p/249194
- Logs https://github.com/tspannhw/nifi-logs
- Beacons, BLE https://github.com/tspannhw/minifi-estimote
- Deep Speech https://github.com/tspannhw/nifi-deepspeech
- Syslog https://www.exabeam.com/siem/apache-nifi-ingest-syslog-data/
- IBM Event Streams https://www.ibm.com/cloud/blog/connecting-ibm-event-streams-to-apache-nifi
- Mark Logic JDBC https://www.marklogic.com/blog/apache-nifi-quickstart/
- Microsoft Teams https://aussie.nifi.ninja/blog/
- Office 365 https://github.com/pontus-vision/pontus-nifi-processors-office365
NiFi Stateless
- https://dev.to/tspannhw/exploring-apache-nifi-1-10-parameters-and-stateless-engine-44n4
- Example with Kafka https://github.com/tspannhw/stateless-examples
- https://blog.cloudera.com/paving-the-pathway-to-cloudera-data-platform-with-cloudera-dataflow/
Machine Learning, Deep Learning, AI
- https://dev.to/tspannhw/easy-deep-learning-in-apache-nifi-with-djl-2d79
- https://dev.to/tspannhw/tracking-air-quality-with-apache-nifi-cloudera-data-science-workbench-pyspark-and-parquet-28c
- https://dev.to/tspannhw/updating-machine-learning-models-at-the-edge-with-apache-nifi-and-minifi-3ama
- https://community.cloudera.com/t5/Community-Articles/Using-Apache-NiFi-with-Apache-MXNet-GluonCV-for-YOLO-3-Deep/ta-p/248979?es_id=10e1da1507
- https://community.cloudera.com/t5/Community-Articles/Apache-NiFi-Processor-for-Apache-MXNet-SSD-Single-Shot/ta-p/249240
- https://community.cloudera.com/t5/Community-Articles/Apache-Deep-Learning-101-Processing-Apache-MXNet-Model/ta-p/247944
- https://community.cloudera.com/t5/Community-Articles/Detecting-Language-with-Apache-NiFi/ta-p/248424
- https://community.cloudera.com/t5/Community-Articles/Steganography-with-Apache-NiFi/ta-p/244728
- Model Servers https://community.cloudera.com/t5/Community-Articles/Using-the-new-MXNet-Model-Server-with-Apache-NiFi/ta-p/247555
- https://github.com/tspannhw/OpenSourceComputerVision
- https://github.com/tspannhw/ApacheDeepLearning202
- https://github.com/tspannhw/DWS2019Barcelona
- https://github.com/tspannhw/ApacheDeepLearning101
- https://community.cloudera.com/t5/Community-Articles/Open-NLP-Example-Apache-NiFi-Processor/ta-p/249293
Best Practices
- https://dev.to/tspannhw/no-more-spaghetti-flows-2emd
- https://dev.to/tspannhw/apache-nifi-load-balancing-via-load-balanced-connections-593m
- https://www.datainmotion.dev/2019/06/performance-testing-apache-nifi-part-1.html
- http://lonnifi.blogspot.com/2019/11/back-pressure-prediction-deep-dive.html
- https://community.cloudera.com/t5/Community-Articles/HDF-CFM-NIFI-Best-practices-for-setting-up-a-high/ta-p/244999
Tutorials
How To
- Connect to Atlas https://dev.to/tspannhw/connecting-apache-nifi-to-apache-atlas-for-data-governance-at-scale-in-streaming-f5n
- sFTP https://community.cloudera.com/t5/Community-Articles/How-to-Retrieve-files-from-a-SFTP-server-using-NiFi-GetSFTP/ta-p/245430?search-action-id=37601510770&search-result-uid=245430
Integration Testing
IoT, Devices, MQTT, Sensors
- https://dev.to/tspannhw/predicting-sensor-readings-with-time-series-machine-learning-3ngc
- https://dev.to/tspannhw/edgeai-google-coral-with-coral-environmental-sensors-and-tpu-with-nifi-and-minifi-updated-efm-oh9
- https://dev.to/tspannhw/analyzing-wood-burning-stoves-with-flank-stack-minifi-flink-nifi-kafka-kudu-36on
- https://github.com/tspannhw/IoTFusion2018Talk
Administration,CLI, REST, Management, DevOps, CI/CD, Security
- https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect
- https://dev.to/tspannhw/apache-nifi-and-nifi-registry-administration-3c92
- https://dev.to/tspannhw/using-nifi-cli-to-restore-nifi-flows-from-backups-18p9
- https://dev.to/tspannhw/nifi-toolkit-cli-for-nifi-1-10-213h
- https://dev.to/tspannhw/backup-and-restore-nifi-registry-templates-14m
- https://nipyapi.readthedocs.io/en/latest/
- https://community.cloudera.com/t5/Community-Articles/How-to-create-user-generated-keys-for-securing-NiFi/ta-p/245551
- https://github.com/tspannhw/BackupRegistry
- https://community.cloudera.com/t5/Community-Articles/DevOps-Tips-Using-the-Apache-NiFi-Toolkit-with-Apache-NiFI-1/ta-p/248376
- https://community.cloudera.com/t5/Community-Articles/Big-Data-DevOps-Apache-NiFi-Flow-Versioning-and-Automation/ta-p/247976
- https://dzone.com/articles/finding-nifi-errors
- https://community.cloudera.com/t5/Community-Articles/Building-a-Custom-Apache-NiFi-Operations-Dashboard-Part-1/ta-p/249060
Quick Starts
- https://github.com/apache/kudu/blob/master/examples/quickstart/nifi/README.adoc
- https://nifi.apache.org/docs/nifi-docs/html/getting-started.html
- https://community.cloudera.com/t5/Community-Articles/Launching-a-NiFi-Docker-Instance/ta-p/248197
- https://hub.docker.com/r/apache/nifi/
- https://docs.cloudera.com/cdf-datahub/7.1.0/quick-start-fm/topics/cdf-datahub-fm-cluster-setup-quick-start-overview.html
Workshops & Crash Courses
- https://github.com/tspannhw/ClouderaFlowManagementWorkshop
- https://github.com/aperepel/nifi-workshop-2019
- https://www.youtube.com/watch?v=fblkgr1PJ0o
- https://github.com/abajwa-hw/masterclass
- https://github.com/simonellistonball/masterclass-hdf
FLaN(ifi)K
Meetups
Videos
- https://www.youtube.com/watch?v=IUjz-rhA3xs
- https://www.youtube.com/user/HadoopSummit/videos
- https://www.youtube.com/channel/UCDIDMDfje6jAvNE8DGkJ3_w?view_as=subscriber
- https://www.youtube.com/user/clouderahadoop
- https://www.youtube.com/watch?v=XsL63ZQYmLE
- https://www.youtube.com/watch?v=xsC4cjSBv8k
- https://www.youtube.com/watch?v=TmdROpjVtM4
- https://www.youtube.com/watch?v=IUjz-rhA3xs
- https://www.youtube.com/watch?v=Tt8TSlHu7PE
- https://www.youtube.com/watch?v=QQks0KAvT58
- https://www.youtube.com/watch?v=QWXXaRE1VUI
Cloud
- https://dev.to/tspannhw/streaming-data-with-cloudera-data-flow-cdf-into-public-cloud-cdp-188b
- https://docs.cloudera.com/cdf-datahub/7.2.0/nifi-hive-ingest/topics/cdf-datahub-nifi-hive-ingest.html
- https://docs.cloudera.com/cdf-datahub/7.2.0/nifi-hive-ingest/topics/cdf-datahub-hive-ingest-service-account.html
- https://docs.cloudera.com/cdf-datahub/7.2.0/nifi-hive-ingest/topics/cdf-datahub-hive-ingest-idbroker-mapping.html
- https://docs.cloudera.com/cdf-datahub/7.2.0/nifi-hive-ingest/topics/cdf-datahub-hive-ingest-data-target.html
- https://www.calculatedsystems.com/post/moving-data-to-the-cloud
- https://medium.com/swlh/operationalising-nifi-on-kubernetes-1a8e0ae16a6c
Building Custom Processors
- https://community.cloudera.com/t5/Community-Articles/Building-a-Custom-Processor-Using-IntelliJ/ta-p/244343
- https://community.hortonworks.com/content/kbentry/116803/building-a-custom-processor-in-apache-nifi-12-for.html
- https://github.com/alopresto/nifi-byop-dws
- https://github.com/DevWorxCo/nifi-starter
- https://github.com/influxdata/nifi-influxdb-bundle
Custom Processors
- https://github.com/All-of-Us-Research-Program/nifi-fhir-processor
- https://github.com/1904labs/nifi-standardize-date-bundle
- https://github.com/bbende/nifi-dependency-example
- https://github.com/tspannhw/ExecuteClouderaML
- https://github.com/minyk/nifi-headlessbrowser-processor
- https://github.com/tspannhw/nifi-djl-processor
- https://github.com/tspannhw/nifi-tensorflow-processor
- https://github.com/tspannhw/nifi-mxnetinference-processor
- https://github.com/tspannhw/nifi-extracttext-processor
- https://github.com/tspannhw/nifi-langdetect-processor
- https://github.com/tspannhw/nifi-attributecleaner-processor
- https://github.com/tspannhw/nifi-convertjsontoddl-processor
- https://github.com/tspannhw/nifi-postimage-processor
- https://github.com/tspannhw/GetWebCamera
- https://github.com/tspannhw/nifi-imageextractor-processor
- https://github.com/tspannhw/nifi-puttwitter-processor
- https://github.com/tspannhw/nifi-tensorflow-processor
- https://github.com/simonellistonball/nifi-OpenCV
- https://github.com/jdye64/nifi-openalpr
- https://github.com/AnkurJais/nifi-bigquery-custom-processor
- https://github.com/dDMarley/nifi-custom-processors
- https://github.com/myagmurs/bitmapprocessor
- https://github.com/linksmart/nifi-customlistentcp-processor
- https://github.com/andrewpalumbo/nifi-processors
- https://github.com/ligaroba/apache-nifi-smpp-processor
- https://github.com/shekhar-kotekar/nifi-word-count-custom-processor
- https://github.com/philenius/sum-json-array-of-numbers-custom-nifi-processor
- https://github.com/bongye/nifi-bitcoin
- https://github.com/ahmednaggar19/nifi-custom-processor-count-record-attribute
- https://github.com/PPcraft/iec104-bundle
- https://github.com/A-Hussien2011/ConvertJsonToHQL
- https://github.com/gudiseva/nifi-kafka-timestamp
- https://github.com/davidglevy/example-nifi-csv-cleaner
- https://github.com/kai5263499/nifi-binarygrpc
- https://github.com/adammichalsky/Hana-Custom-Processor
- https://github.com/cukuzz/nifi-custom-hdfs-processors-example
- https://github.com/jdye64/nifi-addons
- https://github.com/apsaltis/nifi-soap
- https://github.com/jdye64/nifi-gpiorest
- https://github.com/everwatchsolutions/aces-nifi-processors-bundle
- https://github.com/swiftops/nifi-infinispan-processor
- https://github.com/franziska-w/nifi-custom-processor
- https://github.com/wahid18benz/Apache-NiFi-Custom-Processor
- https://github.com/laurencedaluz/nifi-satori-bundle
- https://github.com/mataram/Nifi-NSQ-Consumer
- https://github.com/ORION6194/ApacheNifi-CustomProcessors
- https://github.com/bpenelli/nifi-bpenelli-bundle-1.8.0
- https://github.com/simonellistonball/PiWiNiFi
- https://github.com/tspannhw/nifi-corenlp-processor
- https://github.com/nagarajjayakumar/nifi-hana-custom-processor
- https://github.com/tspannhw/nifi-extracttext-processor
- https://github.com/tspannhw/nifi-nlp-processor
- https://github.com/dormstd/nifi-CKANProcessorBundle
- https://community.cloudera.com/t5/Community-Articles/Apache-NiFi-Processor-Building-a-SQL-DDL-Schema-From-A-JSON/ta-p/247989
- https://github.com/iss-lab/nifi-stanfordcorenlp-processor
- https://github.com/tspannhw/nifi-puttwitter-processor
- https://github.com/compose-ex/nifi-compose-bundle
- https://github.com/BRGM/Nifi-WKB-to-WKT-processor
- https://github.com/nagarajjayakumar/nifi-hana-custom-processor
- https://github.com/dvas0004/nifi-loadbalancer
- https://github.com/dvas0004/nifi-webscraper
Migrate Flume Flows to NiFi
JMS, RDBMS, Kafka, Syslog, Twitter, Parquet, Kudu, Hive, HDFS, REST
- https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-jms-to-x-and-x-to-jms-1g02
- https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-any-relational-database-to-from-anywhere-45d1
- https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-http-rest-sink-and-http-rest-source-to-kafka-sink-1257
- https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-syslog-to-kafka-3a1k
- https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-twitter-source-to-kafka-sink-31d1
- https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-apache-parquet-on-hdfs-1npb
- https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-hdfs-kudu-file-hive-55gj
- Templates https://github.com/tspannhw/flume-to-nifi
FDLC, SDLC, DevOps, Backup, Restore, Migration, CI/CD
- https://www.datainmotion.dev/2019/03/apache-nifi-operations-and-monitoring.html
- https://community.cloudera.com/t5/Community-Articles/How-to-import-a-flow-to-Nifi-registry-in-CDP-Cloud/ta-p/308335
- https://www.datainmotion.dev/2020/09/devops-working-with-parameter-contexts.html
- https://www.datainmotion.dev/2020/10/automating-building-migration-backup.html
- https://www.datainmotion.dev/2020/08/deleting-schemas-from-cloudera-schema.html
- https://www.datainmotion.dev/2019/09/backup-and-restore-nifi-registry.html
- https://www.datainmotion.dev/2019/04/simple-apache-nifi-operations-dashboard.html
- https://www.youtube.com/watch?v=yKmVBTeZS4c
- https://medium.com/@abdelkrim.hadjidj/fdlc-towards-flow-development-life-cycle-with-nifi-registries-82e1ee866fab
- https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/
- https://community.cloudera.com/t5/Community-Articles/NiFi-SDLC-Automation-in-Python-with-NiPyApi-Part-1/ta-p/247693
- https://github.com/Chaffelson/nipyapi
- https://docs.cloudera.com/cfm/2.0.4/cfm-getting-started/topics/cfm-gs-overview.html
- https://leanjavaengineering.wordpress.com/2018/07/26/quick-tip-using-git-with-nifi-registry-in-docker/
- https://github.com/michalklempa/docker-nifi-registry
- https://github.com/steven-matison/NiFi-Templates
- https://github.com/tspannhw/BackupRegistry
- https://github.com/itarano/nifi-pre-pro-registry
- https://github.com/erdrix/nigoapi
- https://blog.cloudera.com/cloudera-flow-management-continuous-delivery-while-minimizing-downtime/
- https://github.com/alopresto/slides
- https://bryanbende.com/development/2017/05/10/apache-nifi-1-2-0-component-versioning
- https://www.infoq.com/presentations/secure-iot-apache-minifi/
- https://biconsulting.hu/letoltes/2020/2020budapestdata/D1_Daniel_Chaffelson_Cloudera.pdf
NiFi Notes
- https://www.javahotchocolate.com/notes/nifi.html
- https://martin.atlassian.net/wiki/spaces/lestermartin/pages/740196355/NiFi+Cheat+Sheet
Webinars
- https://www.brighttalk.com/webcast/18452/479025/demo-jam-live-create-data-flows-with-apache-nifi
- https://www.brighttalk.com/webcast/15417/354772
Integrations
- https://www.stardog.com/labs/blog/stardog-data-flow-automation-with-nifi/
- https://hackernoon.com/data-ingestion-using-apache-nifi-for-building-data-lake-using-twitter-data-1f3d3da6fe1d
CDP Public Cloud, AWS, Azure, GCP
- https://community.cloudera.com/t5/Community-Articles/How-to-use-NiFi-to-write-API-data-to-CDP-CDW/ta-p/306770
- https://www.datainmotion.dev/2020/04/streaming-data-with-cloudera-data-flow.html
- https://www.datainmotion.dev/2020/07/flank-in-cloud-huge-cloudera-data.html
- https://www.datainmotion.dev/2020/12/new-release-for-hdf-352-and-cloudera.html
- https://www.datainmotion.dev/2020/04/streaming-data-with-cloudera-data-flow.html
- https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-kafka-ingest/topics/cdf-datahub-fm-kafka-ingest-overview.html
- https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-hive-ingest/topics/cdf-datahub-nifi-hive-ingest.html
- https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-hbase-ingest/topics/cdf-datahub-nifi-hbase-ingest.html
- https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-kudu-ingest/topics/cdf-datahub-nifi-kudu-ingest.html
- https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-azure-ingest/topics/cdf-datahub-fm-adls-ingest-overview.html
- https://docs.cloudera.com/cfm/2.0.4/site-to-site/topics/cdf-datahub-site-to-site.html
- https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-aws-ingest/topics/cdf-datahub-fm-s3-ingest-overview.html
- https://docs.cloudera.com/cdf-datahub/7.2.6/planning-your-flow-management-deployment/topics/cdf-datahub-fm-cluster-layout.html
- https://medium.datadriveninvestor.com/how-to-build-a-simple-data-flow-with-apache-nifi-1ef3b25fe2c4
MemSQL
- https://www.youtube.com/watch?v=Jnz-a_wCYp4
- https://medium.com/@moha.ajori/using-apache-nifi-to-load-tweets-from-twitter-api-to-memsql-19e19a3be20e
GreenPlum DB
- https://www.cdata.com/kb/tech/greenplum-jdbc-apache-nifi.rst
- https://github.com/DanielePalaia/gpss-nifi
Teradata
- https://www.cdata.com/kb/tech/teradata-jdbc-apache-nifi.rst
- https://community.cloudera.com/t5/Community-Articles/Using-Teradata-JDBC-connector-in-NiFi/ta-p/246783
Microsoft SQL Server (MS SQL)
- https://community.cloudera.com/t5/Support-Questions/Nifi-Connection-to-MSSQL-server-DB/td-p/95642
- https://www.cdata.com/kb/tech/sql-jdbc-apache-nifi.rst
- https://nathanlabadie.com/apache-nifi-ms-sql-and-kerberos-authentication/
- https://gist.github.com/ijokarumawak/42c257afb5e80361e502564085d7999e
- https://medium.com/@danielyahn/nifi-ms-sql-integrated-security-9efc184872db
- https://rapidminernotes.blogspot.com/2018/01/populating-sql-server-from-apache-nifi.html
Oracle
- https://www.cdata.com/kb/tech/oracledb-jdbc-apache-nifi.rst
- https://www.progress.com/blogs/tutorial-access-data-via-jdbc-with-apache-nifi
- https://www.youtube.com/watch?v=XsL63ZQYmLE
- https://www.youtube.com/watch?v=qsIJFcQYp2E
- John Kuchmek at ApacheCon 2020 Incrementally Streaming from Oracle -> https://www.youtube.com/watch?v=nPewLy7Zjkg&list=PLlmpIaH7URGXYTdiHhlYbSkAO_B_RDDn0&index=28
S3 Data Indexing
Cloud
- https://www.batchiq.com/nifi-process-s3-event-notifications.html
- https://gist.github.com/jvwing
- https://www.batchiq.com/process-cloudtrail-events-with-nifi.html
- https://www.batchiq.com/s3-ingest-with-nifi.html
- https://www.batchiq.com/database-extract-with-nifi.html
- https://www.batchiq.com/database-ingest-with-nifi.html
CDC / Database
- https://community.cloudera.com/t5/Community-Articles/Incremental-Fetch-in-NiFi-with-QueryDatabaseTable/ta-p/247073
- https://community.cloudera.com/t5/Community-Articles/Change-Data-Capture-CDC-with-Apache-NiFi-Part-1-of-3/ta-p/246623
- https://github.com/edufer/nifi-cdc-sqlserver
- https://dev.to/tspannhw/simple-change-data-capture-cdc-with-sql-selects-via-apache-nifi-flank-19m4
- https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-cdc-mysql-nar/1.13.2/org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL/
- https://www.linkedin.com/pulse/achieving-incremental-fetch-change-data-capture-via-apache-rajpal/
- https://community.cloudera.com/t5/Community-Articles/PutHive3Streaming-to-do-Basic-CDC-with-NiFi/ta-p/249187
- https://community.cloudera.com/t5/Community-Articles/Simple-Change-Data-Capture-CDC-with-SQL-Selects-via-Apache/ta-p/308376
- https://community.cloudera.com/t5/Community-Articles/Basic-CDC-on-SAP-HANA-using-Apache-NIFI/ta-p/249323
- https://www.qlik.com/us/data-management/nifi
- https://www.marklogic.com/blog/apache-nifi-quickstart/
- https://www.vertica.com/kb/NifiCG/Content/Partner/NifiCG.htm
- https://community.cloudera.com/t5/Community-Articles/Incrementally-Streaming-RDBMS-Data-to-Your-Hadoop-DataLake/ta-p/247927
- https://www.batchiq.com/nifi-flow-examples.html
- https://blog.pythian.com/database-migration-using-apache-nifi/
- https://www.datainmotion.dev/2019/04/oracle-golden-gate-to-apache-kafka-to.html
- https://github.com/anair-it/nifi-docker
- https://medium.com/trendyol-tech/using-apache-nifi-for-migrating-data-fe1c224c7b16
Provenance Checking
Quick Tips: check nifi security policy on provenance, check configuration for repo-type, check for available disk space, unsynced time, restart all servers. Configuration: nifi.provenance.repository.max.storage.time
- https://javasupportzone.blogspot.com/2019/07/possible-reasons-for-processors-data.html
- https://community.cloudera.com/t5/Support-Questions/NiFi-Provenance-Not-Available-in-UI/td-p/179999#:~:text=Re%3A%20NiFi%20Provenance%20Not%20Available%20in%20UI&text=Try%20stopping%20Nifi%20and%20purging,read%2Fwrite%20in%20set%20directory.
- https://stackoverflow.com/questions/41487509/data-provenance-disabled-nifi
Videos
Misc
Other Languages
Legacy / Older Editions
- https://github.com/jfrazee/awesome-nifi
- https://dzone.com/articles/apache-nifi-10-cheatsheet
- https://dzone.com/articles/monitoring-for-apache-nifi-17
- https://www.cdata.com/kb/tech/salesforce-jdbc-apache-nifi.rst
- https://github.com/UKHomeOffice/cdp-nifi-processors
- https://github.com/dstreev/nifi-processor-examples
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent