Code Monkey home page Code Monkey logo

whodap's Introduction

whodap

PyPI version example workflow codecov Code style: black

whodap | Simple RDAP Utility for Python

  • Support for asyncio HTTP requests (httpx)
  • Leverages the SimpleNamespace type for cleaner RDAP Response traversal
  • Keeps the familiar look of WHOIS via the to_whois_dict method for DNS lookups

Quickstart

import asyncio
from pprint import pprint

import whodap

# Looking up a domain name
response = whodap.lookup_domain(domain='bitcoin', tld='org') 
# Equivalent asyncio call
loop = asyncio.get_event_loop()
response = loop.run_until_complete(whodap.aio_lookup_domain(domain='bitcoin', tld='org'))
# "response" is a DomainResponse object. It contains the output from the RDAP lookup.
print(response)
# Traverse the DomainResponse via "dot" notation
print(response.events)
"""
[{
  "eventAction": "last update of RDAP database",
  "eventDate": "2021-04-23T21:50:03"
},
 {
  "eventAction": "registration",
  "eventDate": "2008-08-18T13:19:55"
}, ... ]
"""
# Retrieving the registration date from above:
print(response.events[1].eventDate)
"""
2008-08-18 13:19:55
"""
# Don't want "dot" notation? Use `to_dict` to get the RDAP response as a dictionary
pprint(response.to_dict())
# Use `to_whois_dict` for the familiar look of WHOIS output
pprint(response.to_whois_dict())
"""
{abuse_email: '[email protected]',
 abuse_phone: 'tel:+1.6613102107',
 admin_address: 'P.O. Box 0823-03411, Panama, Panama, PA',
 admin_email: '[email protected]',
 admin_fax: 'fax:+51.17057182',
 admin_name: 'WhoisGuard Protected',
 admin_organization: 'WhoisGuard, Inc.',
 admin_phone: 'tel:+507.8365503',
 billing_address: None,
 billing_email: None,
 billing_fax: None,
 billing_name: None,
 billing_organization: None,
 billing_phone: None,
 created_date: datetime.datetime(2008, 8, 18, 13, 19, 55),
 domain_name: 'bitcoin.org',
 expires_date: datetime.datetime(2029, 8, 18, 13, 19, 55),
 nameservers: ['dns1.registrar-servers.com', 'dns2.registrar-servers.com'],
 registrant_address: 'P.O. Box 0823-03411, Panama, Panama, PA',
 registrant_email: '[email protected]',
 registrant_fax: 'fax:+51.17057182',
 registrant_name: 'WhoisGuard Protected',
 registrant_organization: None,
 registrant_phone: 'tel:+507.8365503',
 registrar_address: '4600 E Washington St #305, Phoenix, Arizona, 85034',
 registrar_email: '[email protected]',
 registrar_fax: None,
 registrar_name: 'NAMECHEAP INC',
 registrar_phone: 'tel:+1.6613102107',
 status: ['client transfer prohibited'],
 technical_address: 'P.O. Box 0823-03411, Panama, Panama, PA',
 technical_email: '[email protected]',
 technical_fax: 'fax:+51.17057182',
 technical_name: 'WhoisGuard Protected',
 technical_organization: 'WhoisGuard, Inc.',
 technical_phone: 'tel:+507.8365503',
 updated_date: datetime.datetime(2019, 11, 24, 13, 58, 35)}
"""

Exported Functions and Classes

Object Description
lookup_domain Performs an RDAP query for the given Domain and TLD
lookup_ipv4 Performs an RDAP query for the given IPv4 address
lookup_ipv6 Performs an RDAP query for the given IPv6 address
lookup_asn Performs an RDAP query for the Autonomous System with the given Number
aio_lookup_domain async counterpart to lookup_domain
aio_lookup_ipv4 async counterpart to lookup_ipv4
aio_lookup_ipv6 async counterpart to lookup_ipv6
aio_lookup_asn async counterpart to lookup_asn
DNSClient Reusable client for RDAP DNS queries
IPv4Client Reusable client for RDAP IPv4 queries
IPv6Client Reusable client for RDAP IPv6 queries
ASNClient Reusable client for RDAP ASN queries

Common Usage Patterns

  • Using the DNSClient:
import whodap

# Initialize an instance of DNSClient using classmethods: `new_client` or `new_aio_client`
dns_client = whodap.DNSClient.new_client()
for domain, tld in [('google', 'com'), ('google', 'buzz')]:
    response = dns_client.lookup(domain, tld)
    
# Equivalent asyncio call
dns_client = await whodap.DNSClient.new_aio_client()
for domain, tld in [('google', 'com'), ('google', 'buzz')]:
    response = await dns_client.aio_lookup(domain, tld)
    
# Use the DNSClient contextmanagers: `new_client_context` or `new_aio_client_context`
with whodap.DNSClient.new_client_context() as dns_client:
    for domain, tld in [('google', 'com'), ('google', 'buzz')]:
        response = dns_client.lookup(domain, tld)

# Equivalent asyncio call
async with whodap.DNSClient.new_aio_client_context() as dns_client:
    for domain, tld in [('google', 'com'), ('google', 'buzz')]:
        response = await dns_client.aio_lookup(domain, tld)
  • Configurable httpx client:
import asyncio

import httpx
import whodap

# Initialize a custom, pre-configured httpx client ...
httpx_client = httpx.Client(proxies=httpx.Proxy('https://user:pw@proxy_url.net'))
# ... or an async client
aio_httpx_client = httpx.AsyncClient(proxies=httpx.Proxy('http://user:pw@proxy_url.net'))

# Three common methods for leveraging httpx clients are outlined below:

# 1) Pass the httpx client directly into the convenience functions: `lookup_domain` or `aio_lookup_domain`
# Important: In this scenario, you are responsible for closing the httpx client.
# In this example, the given httpx client is used as a contextmanager; ensuring it is "closed" when finished.
async with aio_httpx_client:
    futures = []
    for domain, tld in [('google', 'com'), ('google', 'buzz')]:
        task = whodap.aio_lookup_domain(domain, tld, httpx_client=aio_httpx_client)
        futures.append(task)
    await asyncio.gather(*futures)

# 2) Pass the httpx_client into the DNSClient classmethod: `new_client` or `new_aio_client`
aio_dns_client = await whodap.DNSClient.new_aio_client(aio_httpx_client)
result = await aio_dns_client.aio_lookup('google', 'buzz')
await aio_httpx_client.aclose()

# 3) Pass the httpx_client into the DNSClient contextmanagers: `new_client_context` or `new_aio_client_context`
# This method ensures the underlying httpx_client is closed when exiting the "with" block.
async with whodap.DNSClient.new_aio_client_context(aio_httpx_client) as dns_client:
    for domain, tld in [('google', 'com'), ('google', 'buzz')]:
        response = await dns_client.aio_lookup(domain, tld)
  • Using the to_whois_dict method and RDAPConformanceException
import logging

from whodap import lookup_domain
from whodap.errors import RDAPConformanceException

logger = logging.getLogger(__name__)

# strict = False (default)
rdap_response = lookup_domain("example", "com")
whois_format = rdap_response.to_whois_dict()
logger.info(f"whois={whois_format}")
# Given a valid RDAP response, the `to_whois_dict` method will attempt to
# convert the RDAP format into a flattened dictionary of WHOIS key/values

# strict = True
try:
    # Unfortunately, there are instances in which the RDAP protocol is not
    # properly implemented by the registrar. By default, the `to_whois_dict`
    # will still attempt to parse the into the WHOIS dictionary. However,
    # there is no guarantee that the information will be correct or non-null. 
    # If your applications rely on accurate information, the `strict=True`
    # parameter will raise an `RDAPConformanceException` when encountering
    # invalid or incorrectly formatted RDAP responses.
    rdap_response = lookup_domain("example", "com")
    whois_format = rdap_response.to_whois_dict(strict=True)
except RDAPConformanceException:
    logger.exception("RDAP response is incorrectly formatted.")

Contributions

  • Interested in contributing?
  • Have any questions or comments?
  • Anything that you'd like to see?
  • Anything that doesn't look right?

Please post a question or comment.

Roadmap

[alpha] 0.1.X Release:

  • Support for RDAP "domain" queries
  • Support for RDAP "ipv4" and "ipv6" queries
  • Support for RDAP ASN queries
  • Abstract the HTTP Client (httpx is the defacto client for now)
  • Add parser utils/helpers for IPv4, IPv6, and ASN Responses (if someone shows interest)
  • Add RDAP response validation support leveraging ICANN's tool

RDAP Resources:

whodap's People

Contributors

grigi avatar pogzyb avatar rahulxs avatar swordphishgroup avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

whodap's Issues

ASNClient/RDAPClient does not follow redirects

the IANA ASN registry is not fully precise, and some numbers are actually handled by another RIR. In that case, the lookup response represents a 303 "See Other" redirect, which is not handled by whodap

example:

import whodap
whodap.lookup_asn(1104)
---------------------------------------------------------------------------
BadStatusCode                             Traceback (most recent call last)
Input In [28], in <cell line: 1>()
----> 1 whodap.lookup_asn(1104)

File ~/.pyenv/versions/3.9.10/envs/test/lib/python3.9/site-packages/whodap/__init__.py:189, in lookup_asn(asn, httpx_client)
    187 asn_client = ASNClient.new_client(httpx_client)
    188 try:
--> 189     resp = asn_client.lookup(asn)
    190 finally:
    191     # close the default client created by ASNClient if it exists;
    192     # otherwise it's up to the user to close their `httpx_client`
    193     if not httpx_client:

File ~/.pyenv/versions/3.9.10/envs/test/lib/python3.9/site-packages/whodap/client.py:564, in ASNClient.lookup(self, asn, auth_ref)
    562 server = self._get_rdap_server(asn)
    563 href = self._build_query_href(server, str(asn))
--> 564 rdap_resp = self._get_authoritative_response(href)
    565 asn_response = ASNResponse.from_json(rdap_resp.read())
    566 return asn_response

File ~/.pyenv/versions/3.9.10/envs/test/lib/python3.9/site-packages/whodap/client.py:184, in RDAPClient._get_authoritative_response(self, href, depth)
    182 resp = self._get_request(href)
    183 try:
--> 184     self._check_status_code(resp.status_code)
    185 except NotFoundError:
    186     # Occasionally, gTLD RDAP servers are wrong or not fully implemented.
    187     # If we've succeeded previously, but now the href returns a 404,
    188     # return None, so the last OK response is returned.
    189     if depth != 0:

File ~/.pyenv/versions/3.9.10/envs/test/lib/python3.9/site-packages/whodap/client.py:281, in RDAPClient._check_status_code(status_code)
    278     raise RateLimitError(
    279         f"Too many requests: {RDAPStatusCodes.RATE_LIMIT_429}")
    280 else:
--> 281     raise BadStatusCode(f"Status code <{status_code}>")

BadStatusCode: Status code <303>

note that AS1104 is in the Netherlands: https://ipinfo.io/AS1104 so it's registered under RIPE but IANA's registry says the range 1-1876 are under ARIN

New release

Could you please also make a new release for whodap? asyncwhois depends on it, thus it would still pull in asynctest which causes issues on Python 3.11.

Thanks again.

Add code linter workflow

Add a workflow to .github/workflows/ that runs the black code formatter. This will require formatting the current code base. Also, update the README with the code style black badge.

'_build_query_href' method AttributeError

import whodap
whodap.lookup_ipv4('128.201.214.5')

in rare cases, an 'IPv4Address' object is given instead of the 'str' to '_build_query_href' method.

line 444, in _build_query_href
href = posixpath.join(rdap_href, 'ip', target.lstrip('/'))
AttributeError: 'IPv4Address' object has no attribute 'lstrip'

[Proposal] Using same same whois keys

I use this library via your asyncwhois and I have a use case that uses RDAP and Whois at the same time.

  • Query a domain with RDAP
  • Fallback to Whois if RDAP does not support the domain (e.g. example.me, etc.)

In such case, it's very convenient if whodap and asyncwhois use the same keys in the parsed output.
But, currently, there are some diffs. For example,

  • whodap uses techniacl_XXX / asyncwhois uses tech_XXX
  • whodap uses nameserver / asyncwhois uses name_servers

Thus, how about using keys defined in asyncwhois.parse to unify the keys?

Add RDAP conformance checks

Occasionally RDAP servers do not correctly implement the RDAP protocol. Fortunately, there are tools (rdap-conformance-tool) to detect these issues, but they're not easily interoperable with this project.

  1. Add basic checks to to_whois_dict to avoid misleading errors
  2. Explore the possibility of performing jsonschema spec validation leveraging schemas from (rdap-conformance-tool)
  3. Create a new method for test response validity

Related:
#29

TypeError with `to_whois_dict()`

Hello, thanks for creating this library. Let me report an issue with to_whois_dict() method.

import whodap

response = whodap.lookup_domain(domain='namesilo', tld='com')
print(response.to_whois_dict())

This code always fails because of the following TypeError.

Traceback (most recent call last):
  File "/tmp/whodap/test.py", line 4, in <module>
    print(response.to_whois_dict())
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/whodap/whodap/response.py", line 190, in to_whois_dict
    flat.update(self._flat_entities(self.entities))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/whodap/whodap/response.py", line 236, in _flat_entities
    for vcard in entity.vcardArray[-1]:
                 ~~~~~~~~~~~~~~~~~^^^^
TypeError: 'DomainResponse' object is not subscriptable

Probably something wrong with _flat_entities() logic.

Add secureDNS object to `to_whois_dict`

Currently, the to_whois_dict does not extract values related to DNSSEC.

For example, here's a snippet from the output of an RDAP query of google.com:

"secureDNS": {
  "delegationSigned": false
}

This should be captured in the to_whois_dict method as { "dnssec": false }

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.