Comments (6)
Brilliant, cheers Jerome. This sorts our use case out perfectly.
Cracking Python package you've put together here.
from pynonymizer.
To expand on this a little, I'd like to understand the problem being solved here - pynonymizer uses faker to generate the seed data for anonymization, but i'd say the exact way it behaves (and it's subsequent binding to faker) was more of an implementation detail.
To be clear though, i support the case for adding more custom-provided generators - adding support for 3rd party faker providers raises implementation questions like:
- will this be only published, packaged faker providers? e.g. https://pypi.org/project/faker-microservice/
- if it is packaged providers, how do we depend upon the packages (peer-dependencies or sub-dependencies, etc)
- Alternatively, if the user specifies them, do they do it (at the cli or in the strategy file?)
Faker's CLI docs seem to indicate the following usage: https://github.com/joke2k/faker/#command-line-usage
-i {my.custom_provider other.custom_provider} list of additional custom providers to use. Note that is the import path of the package containing your Provider class, not the custom Provider class itself.
Now i'm not against that specifically, but i'd be happier knowing that we were adding a feature that would enable extensibility as a general feature and wouldn't need to track specifically with faker, but i'm curious to hear your thoughts here!
from pynonymizer.
Thanks for the response Jerome.
The problem we're trying to solve is that we have a large database with some non-traditional data located inside a single column. The format of the data looks something like this:
Firstname Lastname <[email protected]>
From what I've seen from the pynonymizer package, there doesn't seem to be a way to filter this column using the strategy.yml
to provide randomised rows based on this format. I could well be wrong here though.
If that's the case, it seems the easiest way for me to achieve this might be to create a specialised Faker provider that I could hopefully point pynonymizer toward and be able to utilise it via a specific fake_type in the strategy.yml
.
Happy for you to tell me there's a much easier way here though!
from pynonymizer.
There was some relevant discussion in #62 about combined fields from faker fields and data consistency. It doesn't look like it'll fit with this use case but it might help indicate where we're at.
As far as dirty workarounds go you could reference data in the seed table in a literal i.e. by concatenating stuff together. But it's gross and definitely brittle. 😅
I agree that the ability to use custom generators might be answer here, it'll also bake in a code interface to custom data formats, and think that can only be a good thing. I'll take a look at this as a feature and update here.
from pynonymizer.
I've added this feature in #75 which should release in 1.21.0
from pynonymizer.
OK so this is out with v1.21.0
! You can check out the docs to see what the expected usage is, but take a look at the mysql integration test also, since I've based this on the usecase here:
a custom provider: https://github.com/jerometwell/pynonymizer/blob/master/tests_integration/mysql/custom_provider.py
referencing that provider in the strategyfile: https://github.com/jerometwell/pynonymizer/blob/master/tests_integration/mysql/sakila.yml
If you could review and close the issue if it's resolved, otherwise we can continue the discussion here 😇
from pynonymizer.
Related Issues (20)
- TypeError: 'NoneType' object is not iterable HOT 5
- Request support for SQLEXPRESS for db-type = mssql HOT 3
- Improve ODBC interaction: Accept a connection string HOT 2
- Improve deployability: Dockerize pynonymizer HOT 2
- Not everything is dumpt HOT 4
- Error on executing command caused by 'ANSI_WARNINGS' for mssql database HOT 8
- Restore stdin / stdout integration test HOT 1
- fix: warning in CI HOT 1
- Error with parameter --skip-steps HOT 2
- handle busy connection when threading HOT 1
- Missing inputs message should not return exit code "0" HOT 2
- Should INPUTS be mandatory when RESTORE_DB is skipped? HOT 2
- PYNONYMIZER_STRATEGY dropped without deprecation HOT 5
- --help output has duplicate --mysql-cmd-opts HOT 2
- Mechanisms to reduce load on server? HOT 5
- Updated docker images? HOT 3
- --verbose flag doesnt work
- Connectionstring fix HOT 2
- pynoymizer doesn't seem to trap mysqldump failing? HOT 3
- How to get PYNONYMIZER_MYSQL_CMD_OPTS to work with socket ?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pynonymizer.