Code Monkey home page Code Monkey logo

auraxium's Introduction

Auraxium

Auraxium is an object-oriented, pure-Python wrapper for the PlanetSide 2 API.
It provides a simple object model that can be used by players and outfits without requiring deep knowledge of the API and its idiosyncrasies.

PyPI - License GitHub Workflow Status Coveralls github branch PyPI - Python Version GitHub Workflow Status CodeFactor Grade PyPI Read the Docs


  • Clean, Pythonic API
  • Asynchronous endpoints to keep apps responsive during high API load
  • Low-level interface for more optimised, custom queries
  • Support for the real-time event streaming service (ESS)
  • User-configurable caching system
  • Fully type annotated

The documentation for this project is hosted at Read the Docs.

Table of Contents

Overview

The Census API used by PlanetSide 2 is powerful, but its design also carries a steep learning curve that makes a lot of basic API interactions rather tedious.

Auraxium streamlines this by hiding the game-agnostic queries behind an object model specific to PlanetSide 2. Whenever data is accessed that is not currently loaded, the library dynamically generates and performs the necessary queries in the background before resuming execution.

All queries that may incur network traffic and latency are asynchronous, which keeps multi-user applications - such as Discord bots - responsive.

Getting Started

All API interactions are performed through the auraxium.Client object. It is the main endpoint used to interact with the API and contains a few essential references, like the current event loop, the connection pool, or the unique service ID used to identify your app.

Regarding service IDs: You can use the default value of s:example for testing, but you may run into rate limiting issues if your app generates more than ~10 queries a minute.

You can apply for your custom service ID here; the process is free, and you usually hear back within a few hours.

Some of these references are also required for any queries carried out behind the scenes, so the client object is also handed around behind the scenes; be mindful when updating them as this may cause issues with ongoing background queries.

Boilerplate code

The aforementioned auraxium.Client object must be closed using the auraxium.Client.close() method before it is destroyed to avoid issues.

Alternatively, you can use the asynchronous context manager interface to automatically close it when leaving the block:

import auraxium

async with auraxium.Client() as client:
    # Your code here

Since Auraxium is an asynchronous library, we also need to wrap our code in a coroutine to be able to use the async keyword.

This gives us the following snippet:

import asyncio
import auraxium

async def main():
    async with auraxium.Client() as client:
        # Your code here

asyncio.run(main())

With that, the stage is set for some actual code.

Usage

The game-specific object representations for PlanetSide 2 can be found in the auraxium.ps2 sub module. Common ones include ps2.Character, ps2.Outfit, or ps2.Item.

Note: The original data used to build a given object representation is always available via that object's .data attribute, which will be a type-hinted, named tuple.

Retrieving Data

The auraxium.Client class exposes several methods used to access the REST API data, like Client.get(), used to return a single match, or Client.find(), used to return a list of matching entries.

It also provides some utility methods, like Client.get_by_id() and Client.get_by_name(). They behave much like the more general Client.get() but are cached to provide better performance for common lookups.

This means that repeatedly accessing an object through .get_by_id() will only generate network traffic once, after which it is retrieved from cache (refer to the Caching section for more information).

Here is the above boilerplate code again, this time with a simple script that prints various character properties:

import asyncio
import auraxium
from auraxium import ps2

async def main():
    async with auraxium.Client() as client:

        char = await client.get_by_name(ps2.Character, 'auroram')
        print(char.name)
        print(char.data.prestige_level)

        # NOTE: Any methods that might incur network traffic are asynchronous.
        # If the data type has been cached locally, no network communication
        # is required.

        # This will only generate a request once per faction, as the faction
        # data type is cached forever by default.
        print(await char.faction())

        # The online status is never cached as it is bound to change at any
        # moment.
        print(await char.is_online())

asyncio.run(main())

Event Streaming

In addition to the REST interface wrapped by Auraxium's object model, PlanetSide 2 also exposes an event stream that can be used to react to in-game events in next to real time.

This can be used to track outfit member performance, implement your own stat tracker, or monitor server population.

The Auraxium client supports this endpoint through a trigger-action system.

Triggers

To receive data through the event stream, you must define a trigger. A trigger is made up of three things:

  • One or more events that tells it to wake up
  • Any number of conditions that decide whether to run or not
  • An action that will be run if the conditions are met

Events

The event type definitions are available in the auraxium.event namespace.

Conditions

Trigger conditions can be attached to a trigger to limit what events it will respond to, in addition to the event type.

This is useful if you have a commonly encountered event (like event.DEATH) and would like your action to only run if the event data matches some other requirement (for example "the killing player must be part of my outfit").

Actions

The trigger's action is a method or function that will be run when the event fires and all conditions evaluate to True.

If the action is a coroutine according to inspect.iscoroutinefunction(), it will be awaited.

The only argument passed to the function set as the trigger action is the event received:

async def example_action(event: Event) -> None:
    """Example function to showcase the signature used for actions.

    Keep in mind that this could also be a regular function (i.e. one
    defined without the "async" keyword).
    """
    # Do stuff here

Registering Triggers

The easiest way to register a trigger to the client is via the auraxium.event.EventClient.trigger() decorator. It takes the event/s to listen for as the arguments and creates a trigger using the decorated function as the trigger action.

Important: Keep in mind that the websocket connection will be continuously looping, waiting for new events to come in.

This means that using auraxium.event.EventClient() as a context manager may cause issues since the context manager will close the connection when the context manager is exited.

import asyncio
from auraxium import event, ps2

async def main():
    # NOTE: Depending on player activity, this script will likely exceed the
    # ~6 requests per minute and IP address limit for the default service ID.
    client = event.EventClient(service_id='s:example')

    @client.trigger(event.BattleRankUp)
    async def print_levelup(evt):
        char = await client.get_by_id(ps2.Character, evt.character_id)

        # NOTE: This value is likely different from char.data.battle_rank as
        # the REST API tends to lag by a few minutes.
        new_battle_rank = evt.battle_rank

        print(f'{await char.name_long()} has reached BR {new_battle_rank}!')

loop = asyncio.new_event_loop()
loop.create_task(main())
loop.run_forever()

Technical Details

The following section contains more detailed implementation details for those who want to know; it is safe to ignore if you are only getting started.

Object Hierarchy

All classes in the Auraxium object model inherit from Ps2Object. It defines the API table and ID field to use for generic queries and implements methods like .get() or .find().

Cache Objects

Cached objects are based off the Cached class, which introduces a class-specific cache for matching instances before falling back to the regular implementation.

It also adds methods for updating the class cache settings at runtime.

See the Caching section for details on the caching system.

Named Objects

Named objects are based off the Named class and always cached. This base class guarantees a .name] attribute and allows the use of the .get_by_name() method, which is also cached.

This caching strategy is almost identical to the one used for IDs, except that it uses a string constructed of the lower-case name and locale identifier to store objects (e.g. 'en_sunderer').

Caching

Auraxium uses timed least-recently-used (TLRU) caches for its objects.

They have a size constraint (i.e. how many objects may be cached at any given time), as well as a maximum age per item (referred to as TTU, "time-to-use"). The TTU is used to ensure frequently used items are updated occasionally and not too far out of date.

When new items are added to the cache, it first removes any expired items (i.e. time_added - now > ttu). It then removes as many least-recently-used items as necessary to accommodate the new elements.

The LRU side of things is implemented via an collections.OrderedDict; every time an item is retrieved from the cache (and is not expired), it is moved back to the start of the dictionary, the last items of the dictionary are then chopped off as needed.

Network Connections

For as long as it is active, the auraxium.Client object will always have a aiohttp.ClientSession running in case the REST API must be accessed.

The websocket connection, which is required for event streaming, is only active when there are triggers registered and active.

If the last trigger is removed, the websocket connection is quietly closed after a delay. If a new trigger is added, it will automatically be recreated in the background.

Object Model Alternatives

For some users or applications, Auraxium's object model may be a bad fit, like for highly nested, complex queries or for users that are already familiar with the Census API.

Here are a few Python alternatives for these cases:

  • The URL generator used by Auraxium to generate the queries for the object model can also be used on its own.

    This still requires some understanding of the Census API data model but takes away the syntactic pitfalls involved.

    It only generates queries, so you will have to pick your own flavour of HTTP library (like requests or aiohttp) to make the queries.

    """Usage example for the auraxium.census module."""
    from auraxium import census
    
    query = census.Query('character', service_id='s:example')
    query.add_term('name.first_lower', 'auroram')
    query.limit(20)
    join = query.create_join('characters_online_status')
    url = str(query.url())
    
    print(url)
    # https://census.daybreakgames.com/s:example/get/ps2:v2/character?c:limit=20&c:join=characters_online_status

    Refer to the census module documentation for details.

  • For an even simpler syntax, you can check out spascou/ps2-census, which was inspired by an earlier version of Auraxium.

    It too sticks closely to the original Census API, but also provides methods for retrieving the queried data.

    It also features a query factory system that allows creation of common queries from templates.

    """Usage example for spascou's ps2-census module."""
    import ps2_census as ps2
    
    query = ps2.Query(ps2.Collection.CHARACTER, service_id='s:example')
    query.filter('name.first_lower', 'auroram')
    query.limit(20)
    query.join(ps2.Join(ps2.Collection.CHARACTERS_ONLINE_STATUS))
    
    print(query.get())
    # {'character_list': [...], 'returned': 1}

    Refer to the ps2-census documentation for details.

Contributing

If you have found a bug or would like to suggest a new feature or change, feel free to get in touch via the repository issues.

Please check out CONTRIBUTING.md before opening any pull requests for details.

auraxium's People

Contributors

lcwilliams avatar leonhard-s avatar lordflashmeow avatar nlioc4 avatar sonst-was avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

auraxium's Issues

Note GainExperience quirks in documentation

(note: this issue is referring to the documentation at https://auraxium.readthedocs.io/en/latest/api/payloads.html#events, not the github or code documentation)

Some GainExperience events do not look quite how you might expect.

For example, grenade assist experience events (IDs 550-555) unintuitively have character_id = killing player & other_id = player killed. There can be multiple events sent for a single kill if the player killed was afflicted by multiple kinds of grenades. I don't know if there are other kinds of experience events that behave like this.

It'd be useful to have a doc page listing what character_id and other_id pertain to for each experience ID, or at least the weird or non-obvious cases. Perhaps group them by character_id-other_id combinations: killer-killed, player-assisted teammate, no other_id etc.

Could also note somewhere experience events that do not seem to actually get sent.

Fix unit test workflow for forks

Forked repositories do not have access to the unencrypted GitHub secrets, which causes the unit tests to fail.

Organisations and private repositories can pass these secrets along, but public repositories currently cannot, so we will have to skip the tests that require secrets on forked repositories.

Add image retrieval utilities

Currently, the only way to use the images is to concatenate the URL yourself using the Ps2Object.data keys. The image-related keys image_id, image_set_id, and image_path are duplicated for every instance.

It would make sense to move these image-specific keys to a separate mix-in class. This would also reduce the attribute count for the affected data classes.

Additionally, this class could hold information on how to retrieve the image/URL and inject its attribute information into the other classes via Sphinx (as it will follow inheritance trees).

Example mix-in class:

@dataclasses.dataclass(frozen=True)
class HasImage:
    """Mix-in class for data types with linked image assets.

    Lots of detailed docs about using images can go here, and
    the attribute documentation below will be included for
    inheriting classes thanks to Sphinx/autodoc.
    
    Attributes:
        image_id: The unique ID of the image asset.
        ...
    
    """

    image_id: int
    image_set_id: int
    image_path: str

Regular dataclass when using the mix-in:

@dataclasses.dataclass(frozen=True)
class AchievementData(Ps2Data, HasImage):

    achievement_id: int
    item_id: int
    ...
    # image_set_id: int
    # image_id: int
    # image_path: str

    @classmethod
    def from_census(cls, data: CensusData) -> 'AchievementData':
        return cls(
            # NOTE: Image assets provided first as the parent class is
            # populated first
            int(data['image_set_id']),
            int(data['image_id']),
            str(data['image_path']),
            # Regular attributes provided below
            int(data['achievement_id']),
            int(data['item_id']),
            int(data['objective_group_id']),
            int(data['reward_id']),
            bool(int(data['repeatable'])),
            LocaleData.from_census(data['name']),
            LocaleData.from_census(data['description']))

Issues with the example code

import asyncio
import auraxium
from auraxium import ps2

async def main():
    async with auraxium.Client() as client:

        char = await client.get_by_name(ps2.Character, 'SYSTEMICSHOCK')
        print(char.name())
        print(char.data.prestige_level)

        # NOTE: Any methods that might incur network traffic are asynchronous.
        # If the data type has been cached locally, no network communication
        # is required.

        # This will only generate a request once per faction, as the faction
        # data type is cached forever by default.
        print(await char.faction())

        # The outfit data type is only cached for a few seconds before being
        # required as it might change.
        outfit = await char.outfit()
        print(outfit.name())

asyncio.run(main())

There are some issues with the example code

  1. Typos that i fixed in my example above
    1.1 NameError: name 'character' is not defined
    1.1 'CharacterData' object has no attribute 'asp_rank'

  2. 2 AttributeError: module 'asyncio' has no attribute 'run_until_complete'

  3. The example character you provided as an example is not part of an outfit and thus outfit is None, thus outfit.name() raises an Exception. I added a character that is in an outfit in my example.
    AttributeError: 'NoneType' object has no attribute 'name'

  4. While the first warning regarding the default service ID makes sense, I also get the following warning that you might want to investigate further.

src/auraxium/auraxium/census/urlgen.py:43: UserWarning: The default service ID is heavily rate-limited. Consider applying for your own service ID at https://census.daybreakgames.com/#devSignup
  warnings.warn('The default service ID is heavily rate-limited. '
SYSTEMICSHOCK
1
Vanu Sovereignty
src/auraxium/auraxium/base.py:145: UserWarning: Unexpected keykeys in payload: []
Please report this error as it hints at a mismatch between the auraxium object model and the API.
  warnings.warn(
  1. The last line of the example code doesn't actually run because of the previous Exception.

Cannot pass conditions with @client.trigger()

When using the @client.trigger(EventType.DEATH) (or any other trigger), you cannot pass a list of conditions to filter the websocket events. Here's your event streaming example, with the simplest condition I could find:

loop = asyncio.get_event_loop()

def example_condition(payload):
    if payload['world_id'] == "1":    # I know you can use the world filter via subscription, but this is a simple filter that allows many results
        return True
    return False

async def main():
    @client.trigger(auraxium.EventType.BATTLE_RANK_UP, conditions=[example_condition])
    async def print_levelup(event):
        char_id = int(event.payload['character_id'])
        char = await client.get_by_id(ps2.Character, char_id)
    
        # NOTE: This value is likely different from char.data.battle_rank as
        # the REST API tends to lag by a few minutes.
        new_battle_rank = int(event.payload['battle_rank'])
    
        print(f'{await char.name_long()} has reached BR {new_battle_rank}!')

loop.create_task(main())
loop.run_forever()

This raises the error

  File "virtualenvs/ps2elo-VDVCoL-1/lib/python3.9/site-packages/auraxium/event.py", line 680, in trigger
    trigger = Trigger(event, *args, name=name, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'conditions'

In the init for Trigger, there's no way to provide conditions, except after initializing the object. As far as I know, it's not possible to modify the trigger object while also using the decorator.

auraxium/auraxium/event.py

Lines 267 to 288 in 5a80345

def __init__(self, event: Union[EventType, str],
*args: Union[EventType, str],
characters: Optional[
Union[Iterable['Character'], Iterable[int]]] = None,
worlds: Optional[
Union[Iterable['World'], Iterable[int]]] = None,
action: Optional[
Callable[[Event], Union[None, Awaitable[None]]]] = None,
name: Optional[str] = None,
single_shot: bool = False) -> None:
self.action = action
self.characters: List[int] = (
[] if characters is None else [c if isinstance(c, int) else c.id
for c in characters])
self.conditions: List[Union[bool, Callable[[CensusData], bool]]] = []
self.events: Set[Union[EventType, str]] = set((event, *args))
self.last_run: Optional[datetime.datetime] = None
self.name = name
self.single_shot = single_shot
self.worlds: List[int] = (
[] if worlds is None else [w if isinstance(w, int) else w.id
for w in worlds])

Unless I'm missing something obvious that hasn't been documented, it seems like the easiest thing to do would be something like:

def __init__(self, ..., conditions: List[Union[bool, Callable[[CensusData], bool]]] = None, ...):
    ...
    self.conditions: List[Union[bool, Callable[[CensusData], bool]]] = conditions if conditions is not None else []
    ...

I'm happy to make a PR to fix this if you want.

Uncaught exception on await Character.items(): KeyError

Bug

Exception raised when trying to retrieve a character's items

Current Behavior:

Exception when awaiting a character's items

Expected Behavior:

The function should return the requested character's items

Steps To Reproduce:

Steps to reproduce the behavior:

  1. Set up the auraxium client
  2. Get a character by name or id
  3. Try to do await char.items()

Code example

import auraxium
from auraxium import ps2
import asyncio

async def main():
    async with auraxium.Client() as client:

    char = await client.get_by_name(ps2.Character, 'ElReyZero')
    items = await char.items()
    print(items)

asyncio.run(main())

Full Traceback:

Traceback (most recent call last):
File "d:\Users\User\Documents\testing.py", line 29, in
asyncio.run(main())
File "D:\Users\User\AppData\Roaming\Python\Python310\lib\asyncio\runners.py", line 44, in run
return loop.run_until_complete(main)
File "D:\Users\User\AppData\Roaming\Python\Python310\lib\asyncio\base_events.py", line 641, in run_until_complete
return future.result()
File "d:\Users\User\Documents\testing.py", line 11, in main
print(await char.items())
File "D:\Users\User\AppData\Roaming\Python\Python310\lib\site-packages\auraxium_proxy.py", line 168, in flatten
return [e async for e in self]
File "D:\Users\User\AppData\Roaming\Python\Python310\lib\site-packages\auraxium_proxy.py", line 168, in
return [e async for e in self]
File "D:\Users\User\AppData\Roaming\Python\Python310\lib\site-packages\auraxium_proxy.py", line 153, in anext
await self._poll()
File "D:\Users\User\AppData\Roaming\Python\Python310\lib\site-packages\auraxium_proxy.py", line 71, in poll
list
= self._resolve_nested_payload(payload)
File "D:\Users\User\AppData\Roaming\Python\Python310\lib\site-packages\auraxium_proxy.py", line 129, in resolve_nested_payload
data.extend(resolve_join(join, parent))
File "D:\Users\User\AppData\Roaming\Python\Python310\lib\site-packages\auraxium_proxy.py", line 106, in resolve_join
value = element[f'{on
}join{join.data.collection}']
KeyError: 'item_id_join_item'

3.11 Traceback: https://pastebin.com/kpz1SpWL

Environment:

  • OS: Windows 11
  • Python 3.10.2
  • Auraxium Version 0.2.2

Create outfit member join/leave event

The trigger system gives us some redundancy to define custom events aside from the ones defined on Daybreak's side. One useful custom event for outfit bots would be to detect members joining or leaving the outfit.

This could easily be done by polling the outfit_member collection and emitting the event if a change is detected.

Alternatively, players might show up as being part of an outfit via the capture/defence experience ticks - this would have to be tested.

Warn when encountering unexpected API fields

The object model currently raises error when expected keys are not present in a given payload dictionary received. This is fine since the current object representation is closely tied to the payload itself.

However, if a new field is added (as was done for ASP rank with ps2/characters and the prestige_level field), the object model currently quietly ignores the extraneous key.

Potential solution

We could change the auraxium.base.Ps2Data.from_census() method(s) in the object model to use dict.pop() rather than direct member access or dict.get(). That way, any expected keys are consumed, and any leftovers are new, fresh, and might provoke some form of UnexpectedPayloadWarning.

Things to look out for:

  • We might want to use a copy of the source dictionary when .pop()-ing out keys, since that payload might be reused elsewhere
  • This also flags joins as "unexpected keys". We could filter out keys with _join_ in them, since the object model never creates any custom-named joins

Faction image_path missing

Hi there👋🏻

In the tests API response for faction.json I can see that image_path should be present however, I'm not receiving a value for it from the API.

Can anyone replicate this?

My code:

    async with auraxium.Client() as client:
        # Get Character object
        player = await client.get_by_name(auraxium.ps2.Character, "Player")
        faction = await player.faction()
        print(faction.data)
        # image_path is not present in the data.

The response data missing image_path:

faction_id=1 name=LocaleData(de='Vanu-Souveränität', en='Vanu Sovereignty', es='Soberanía Vanu', fr='Souveraineté Vanu', it='Sovranità Vanu') code_tag='VS' user_selectable=True

Improve error message when pydantic validation fails

When pydantic encounters an invalid payload, the following exception is raised (example taken from #50):

auraxium\event\_client.py:167> exception=ValidationError(model='MetagameEvent', errors=[{'loc': ('experience_bonus',), 'msg': 'value is not a valid integer', 'type': 'type_error.integer'}])>

This being an error is of no benefit to the user as this is due to a model mismatch in the library and cannot be fixed by the user.
Instead, we should ignore these payloads with a warning and ask the user to submit an issue so the model can be fixed.

Example of obtaining a player's overall kills and deaths.

Hello,

I am quite confused on how I would go about obtaining a player's overall kills, deaths and KDR.

I am unsure what method I need to call on Character in order to obtain these stats.

Would I use stat() or stat_by_faction() or stat_history() ?

Appreciate any help or guidance you can give.

Separate data classes from object model

Currently, the "active" classes (e.g. Character, Outfit) are tightly associated with their respective data class (CharacterData and OutfitData respectively).

This seemed like a promising idea initially, but it is already starting to break down with relational data types, which will only worsen as generic sub-query interfaces (#15) or reports (#18) are implemented. There simply are many types of server responses that do not match a collection.

It seems like moving all of these data classes into a separate models module would be a good solution to underline this difference, and it would greatly declutter some of the wordier modules like ps2.fire as well.

('experience_bonus',), 'msg': 'value is not a valid integer', 'type': 'type_error.integer'}])>

Hi, I'm using a slightly modified version of a snippet of code from the README.

I have altered: @client.trigger(event.BattleRankUp) to @client.trigger(event.MetagameEvent).

Here is my code:

import asyncio
from auraxium import event, ps2

async def main():

    # Initialise event streaming client to the Planetside 2 Census API
    client = event.EventClient(service_id="s:example")

    # Register client trigger
    @client.trigger(event.MetagameEvent)
    async def show_event(evt):
        print(repr(evt))
        print(evt)


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.create_task(main())
    loop.run_forever()

However, the following error occurs on when an event is received:

auraxium\event\_client.py:167> exception=ValidationError(model='MetagameEvent', errors=[{'loc': ('experience_bonus',), 'msg': 'value is not a valid integer', 'type': 'type_error.integer'}])>

Is my usage incorrect?

[FEATURE] Adding tests

Since the Census API might change at any moment (and break at any moment...), it could be very useful to have tests to ensure everything is working. Either to fix things on our side, or to warn the devs that they broke something.

[BUG] Wrong separator for inner joins

Describe the bug
Inner joins are currently all separated with parenthesis: c:join=join1(join1a)(join1b)

Expected behavior
Inner joins should be separated with a comma : c:join=join1(join1a, join1b)

WebSocket version migration

In recent versions of Python, the SSL certificate bypass hack from #55 no longer works, and the default context used can even prevent successful connections entirely.

As a hotfix, the bypass will be disabled in the next version. This should not cause any interruptions to clients as the certs are still good for a few more months (and maybe will be updated this time, rather than expiring).

A more comprehensive fix will follow in conjunction with a migration to the new websockets API. Auraxium is currently using the legacy client-based interface for connections, and any SSL context customization required is tied to it, so might as well do both rather than having to re-do it in a few months anyway.

Add call-soon utility

When working with websocket data, it is common to have a barrage of similar responses sent to the REST API all the time, like when resolving experience IDs or character names.

For these time-insensitive use-cases, it would be better to bundle these requests depending on their URL and query as part of a single, larger query.

A few notes on implementation:

  • Total query size limit (i.e. number of items in the query or length of the query string)
  • Age limit (i.e. no more than 5 seconds delay, oldest items triggers more recent ones as well)
  • This could be extended to also integrate joins, so this must be done at the auraxium.census.Query level (as opposed to the object model or HTTP level)

There were plans for the proxy system to take on similar capabilities at one point. We should make sure this does not overlap with the new proxy system (#22) too much.

Add Event subclasses

Currently, there is a single Event class representing any payloads returned by the websocket API.

It would be beneficial to return a custom data class for each of the ~20 event types. I do not want to replace the EventType enum values as listing the subclasses defined for a given base is more hassle than just looking at the enum values.

Consolidate object model sub modules

As mentioned in #23, there currently exist a large number of tiny modules; only defining 1 or 2 classes each.

These should be grouped together thematically, as was already done for directive-related classes (ps2.directive.py) or the weapon firing mechanics (ps2.fire.py). Similar grouping must be performed for other modules to get them to a sensible length, both for the classes in auraxium.ps2, and auraxium.models (since the two are tightly linked).

player.events_grouped() error.

Hi,

I just ran into an issue using the library.

The following code produces the error:

payload = await run_query(query, session=self._client.session)
line 106, in run_query
data: CensusData = await response.json()

aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: ', url=URL('https://census.daybreakgames.com/s:example/get/ps2:v2/characters_online_status?character_id=id

My code:

import auraxium
import asyncio

async def main():
    async with auraxium.Client() as client:
        player = await client.get_by_name(auraxium.ps2.Character, "character")
        print(await player.events_grouped())

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Detecting API maintenance

Auraxium does currently not provide useful errors if the API is undergoing maintenance.

This is due to the Census API not responding normally but instead redirecting the request to other sites, including:

As neither of these sites exists since the game left SOE, this causes a lower-level error like aiohttp.InvalidURL: or aiohttp.ClientConnectorError:, neither of which is unique to this error state.

Potential solutions

  • It might be possible to deny redirects in as part of the aiohttp.ClientSession.get() call here and provoke another error that way that we can catch.
  • If we could get a aiohttp.ClientResponse instance for the failed request, we could use its history to detect the redirect to a Sony domain.
  • Keep the generic errors mentioned above and parse their URL for station.sony.com, which generally points towards maintenance taking place (still a tiny risk of masking real errors).

Support third-party fallback endpoints

Members of the PS2 API developer community have created fallback endpoints to provide more accurate/reliable data in case the API itself falls behind game updates or has other issues.

With some minor tweaks to the RestClient and EventClient classes, it should be straightforward to support any combination of third-party endpoints as long as they are fully compatible with the Census API formats.

Should be reasonably simple to set up once #56 is done.

Object model discussion

The original object model was very similar to the native API tables/collections.

This makes using the wrapper easier for people used to the Census API's structure, but the entire point of the wrapper was to abstract its clumsy tables into a simple system any player can make sense of without digging through piles of documentation.

The differences will be slight for most common objects (characters, weapons, worlds, etc.), but some tables, more abstraction might be better. Some examples:

  • The "loadout" and "profile" collections are obscure and do not reflect what these names mean in the game
  • The entire weapon damage model is very convoluted in the API (item to weapon, weapon to fire groups, fire groups to fire modes, fire modes to projectiles, projectiles to effects, ...)
  • Renaming "world" to Server and "zone" to Continent would be helpful
  • The continent/facility/region/hex system is hard to get up and running with
  • All of the different statistics tables are hard to parse, and they all come with their own gotchas and traps

Is this a step we want to make? The existing Query interface would always remain available for users that seek lower-level access or need features that cannot be easily implemented via objects.

Add report system

A lot of common operations, like calculating a basic statistic overview for a player, is fairly inefficient under the current object model.

It therefore makes sense to create helper methods that use a single, highly optimised query to retrieve all this data at once, then format it nicely before returning it to the user.

These helpers shall be called "reports" internally for the time being.

To do

  • Compile a list of common reports players might be interested in (add suggestions below)
  • Query other stat tracker developers about how they calculate these statistics (findings go into the wiki for public reference)
  • Find the cheapest query containing all the necessary data
  • Implement the reports themselves

Side note

It was planned to provide an intermediate endpoint between the low-level auraxium.census module and the high-level object model. Maybe reports could be this endpoint in the form of a Report base class?

Can't add terms to join

There is no way to add terms to a join.

In the constructor, it says:

# Additional kwargs are passed on to the `add_term` method
        self._terms: List[Term] = []
        _ = [Term(k.replace('__', '.'), kwargs[k]) for k in kwargs]

But the terms are not added to any list, and there is no add_term method in Join.py.

The only available method is terms(), which appears to expect a list of terms, but offers no way to convert the strings to terms.

def terms(self, *args: Term) -> 'Join':
        """Apply the given list of terms to the join."""
        self._terms = list(args)
        return self

One option is to mimic Query.py and implement the same add_terms() and tweak the Join constructor so it accepts terms as kwargs.

On a side note, why is it self._terms for Join and self.terms for Query?

I just wanted to get some hear if there were some other thoughts before making a PR.

Find syntax for object-specific queries

Some objects (like ps2.Character or ps2.Oufit) require an endpoint that lets the user query related information in a fairly dynamic manner. Good examples are ps2.Character.directives(), which cannot easily be abstracted through objects.

We therefore need a standard way of dealing with these relational tables in an intuitive fashion - the current implementations are placeholders that only redirect kwargs to a census.Query.

I am mostly unsure about the syntax to use - it should be easy to use and read, work for any such relational table, and ideally it would follow the same patterns and kwargs as client.get() or client.find().

Syntax suggestions or other feedback welcome!

Move MaintenanceError check inside the backoff system

Since one of the recent API restarts/changes, these redirect errors are now significantly more common than before (possibly one of the endpoints the load balancer forwards requests to went bad?)

Either way, these errors now are also intermittent, so the code checking for and handling them must also be included in the auto-back off system.

This also warrants checking to see if these redirects work now, which would mean that #14 must be reopened. The new solution might be to just back off up to 5 seconds before giving up and reporting the API as unreachable.

Centralise backoff routine

The exponential backoff system is currently only supported for object-model queries handled by the main client.

This logic should be moved outside the request module and extended to include other use-cases such as the event stream's websocket reconnect attempts.

README example is raising RuntimeError

Not really relevant for real use, but might be impactful for new users as it's a basic example from the README

Reproduction steps:

Config: Python 3.10.4, auraxium 0.2.2, windows 10.

Running the following script as main.py from the github README:

import asyncio
import auraxium
from auraxium import ps2

async def main():
    async with auraxium.Client(service_id="REDACTED") as client:

        char = await client.get_by_name(ps2.Character, 'auroram')
        print(char.name)
        print(char.data.prestige_level)

        # NOTE: Any methods that might incur network traffic are asynchronous.
        # If the data type has been cached locally, no network communication
        # is required.

        # This will only generate a request once per faction, as the faction
        # data type is cached forever by default.
        print(await char.faction())

        # The online status is never cached as it is bound to change at any
        # moment.
        print(await char.is_online())

asyncio.run(main())

Expected result:

The script is working, data from the api is properly displayed => OK

Unexpected result:

RuntimeError raised during the cleanup phase:

Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000016243D50280>
Traceback (most recent call last):
  File "[REDACTED]\Python310\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "[REDACTED]\Python310\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "[REDACTED]\Python310\lib\asyncio\base_events.py", line 750, in call_soon
    self._check_closed()
  File "[REDACTED]\Python310\lib\asyncio\base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Workaround:

Adding asyncio.sleep(2) at the end of the main function fixes the problem, I assume the connection is not closed properly before the loop is closed.

weapon datasheet 'max' and 'min' damage are meaningless due to API quirks

Apologies, I'm too rusty to fix this myself, though I can find the right values. Take this as a suggestion/warning to others trying to use this for damage calculation tools as much as a bug.

Important: This may only apply to infantry weapons, as I haven't looked past that.

Introduction:

Lets say we're looking at the NS-11A. The weapon datasheet from auraxium yields this snippet:
damage=143 damage_min=75 damage_max=225

That can't be right.

Lets look at the ps2 wiki query for the NS11-A's page

The damage ranges have the same quirk. still 143, 75, 225. This is the same across all the infantry weapons I've looked at. Reddit knows the API way better than I do, most of my knowledge of it is from stumbling through them.

Correct values

But when we look under firemodes, each firemode does have the correct values

"max_damage": "143",
"max_damage_range": "10",
"min_damage": "125",
"min_damage_range": "65",

The weapons with different damage per firemode (underbarrel grenade launcher, godsaw, etc) will be edge cases, this can likely be preëmpted by using the description field, and only grabbing semi-auto

Proposed directions:

Ideally, fixing these as the default, or at least making a flag for it.

In my experience, the non-recoil stats most important for comparing weapons on paper are:
max_damage
max_damage_range
min_damage
min_damage_range
fire_rate_ms (time per bullet, not bullet per time as the wiki goes)

Having a method that just outputs these, would help a lot for people new to programming to be able to make tools to compare infantry weapons on paper, instead of the spreadsheets they currently use.

Thanks again for making this! The API is labrynthine and I know everyone is busy, this serves as much as a heads up as an issue.

Improve payload parsing errors

When an error is encountered when parsing a payload, the root converter function' (e.g. int(<something>'s) ValueError propagates out to the user.

Instead, this should be wrapped in a neat PayloadError informing the user that the payload encountered could not be parsed.

Add field properties to object model

Currently most data received through the object model is only available through Ps2Object.data, which is a named tuple containing the original data received.

However, all commonly used data should also be available through class @property-s. The named tuple is only for reference when making requests.

Some thoughts/tentative guidelines:

  • The property should return the most intuitive type possible (i.e. datetime.datetime for timestamps, seconds as float for any durations, etc.)
  • The docstring must include the way the data was generated

Example:

# auraxium.ps2.Character class

@property
def playtime(self) -> float:
    """Return the character's playtime in hours.

    This uses :attr:`Character.data.times.minutes_played`, divided
    by 60.
    """
    return self.data.times.minutes_played / 60.0

Allow unsecured connections for expired API certs

The connection certs for the API server expired three times in the last few years, resulting in Auraxium not connecting at all.
Since the PS2 API event stream and any fetched data are both publicly available, insecure connections are fine for most apps.

A flag should be added that falls back to insecure connections if the original certification check fails. The default behaviour will still be use use secure connections.

Upload latest version to PyPi

The latest version on PyPi was uploaded on October 16, 2019. Do you have the ability to upload the latest version, with the bugfixes we've made?

Separate REST and Streaming API client definition

Currently, all API interactions are handled by the same client instance. In the interest of simplicity, it is worth exploring whether their functionality could be split or at least moved to different definitions (this would also allow making the websocket endpoint an optional dependency).

In the latter case, the EventClient would subclass the regular Client class to extend it with websocket-specific functionality.

I cannot think of a plausible use-case in which the event stream would be required on its own, without any need for REST API access.

Find endpoint for injecting hard-coded fallback data

Several collections are incomplete, such as ps2/experience or ps2/loadout. It would be beneficial to have an endpoint as part of the object model that allows hard-coding certain dummy payloads to specific IDs.

For example, if a user attempts to retrieve a Loadout instance and a NotFoundError is raised, the API wrapper could consult its fallback table and return a dummy payload corresponding to the entries below:

loadout_id profile_id faction_id code_name
28 190 4 NSO Infiltrator
29 191 4 NSO Light Assault
30 192 4 NSO Medic
31 193 4 NSO Engineer
32 194 4 NSO Heavy Assault
45 252 4 NSO MAX

This would fix the error for the user despite the bad API data, but still allows falling back to the API data if it becomes available in the future.

Update Wiki contents after project rewrite

The Wiki is still refering to the original, pre-rewrite version of Auraxium. This should be addressed.

Additional things to add to the Wiki:

  • [ ] auraxium.census Tutorial (Moved to RtD; together with the new API primer everything should be covered)
  • Refreshed API primer
  • Add list of known API error messages and codes
  • Add mapping of metagame event IDs to zone IDs

Reconsider proxy system

The current placeholder proxy objects work, but do not supported nested operations. It'd be very neat if they did.

Example for such a currently unsupported interface:

outfit = await ps2.Outfit.get_by_tag('<your_tag_here>', client=auraxium.Client())

# This is now a proxy object, awaiting it would return a list of OutfitMember instances
members = outfit.members()

# The character attribute failed over to the OutfitMember class and this is now a
# SequenceProxy of Character, with the URL dynamically updated in the background.
characters = outfit.members().character()

Supporting this in parallel to the existing syntax (which I would like to keep) either requires a revamp of how proxy objects work, or some decorator-infused Descriptor object that allows accessing the URL used to link these related object types.

Client.get_by_name not using name.first_lower

Hello,

For the character collection, is there a reason Client.get_by_name is using a query with case(False) on name.first instead of a simple query on name.first_lower:

query.case(False).add_term(field='name.first', value=name)

While the docstring for case mentions that case-insensitive look-ups are significantly slower and should be avoided when possible, citing the character collection as an example?

lowercase field like ``ps2:v2/character.name.first_lower``.

[FEATURE] Implementing show as a named parameter for query and join

Is your feature request related to a problem? Please describe.
I would like to pass a list of fields to show into my query without having to call the show() function.

Describe the solution you'd like
Adding show to the initializer of Query and Join of the type List[str]

Describe alternatives you've considered
Passing in show as a Term does not add the command.
Additionally, hide_fields() (or set_hide_fields() depending on if the PR goes through) takes fields after the first as *args instead of a list, so it is more challenging to set the fields if you already have a list.

Additional context
Add any other context or screenshots about the feature request here.
I'll add to the PR soon.

Add unit tests for object model

With the internal restructuring going on in the models branch right now (065a0e9 and following), unit tests for the object model should be added before merging to ensure compatibility.

Housekeeping:

  • Update the existing test cases to ensure full coverage:
    • cache_test.py
    • census_test.py
    • query_test.py
  • Set up a custom service ID for repository CI/CD as not all things can be tested offline

Object model tests:

  • Add a separate set of tests for the upcoming models submodule. These will be run regularly to ensure the object model matches the API, without having to wait for users to find errors.
  • Add event streaming test cases
  • Add proxy system test cases
  • Add online test cases for common operations. This will require some slow-down to be a good API citizen.

Bonus points:

  • Add a test coverage checker through GitHub actions
  • Add a test coverage status badge

Improve profiling information

Additional stats and metrics to provide to the user to allow them to judge the health of their application. This has no set date or priority yet, just collecting ideas for now.

Rest API

  • Requests per second
  • Requests in the last minute
  • Average latency
  • Latency deviation (API likes to throw in some spikes here and there)
  • Max retries in the last 5 minutes
  • Query retry frequency

Cache Debug

  • Cache size
  • Cache usage
  • Avg. time in cache
  • Proportion of timeouts vs. pushouts

Object Model

  • Caching ratio per datatype (i.e. how many instances go through to the API vs. are restored from cache)

Move data classes to pydantic

Turns out type-enforcing data classes are a thing that already exists.

The move to pydantic should clean the models up a fair bit, too. I'll take a closer look at it ASAP, this should be dealt with before any other shenanigans get finalised.

Reconsider object model getter helpers

Relational tables like characters_item are currently exposed via a provisional interface that effectively wraps a census.Query object with an anonymous **kwargs annotation. This makes them difficult to use, while still not featuring the full join capabilities of the underlying query.

Since this system was introduced, the URL generator has been promoted to be part of the main API. It would therefore be possible to just turn these into query factories, then have the user perform the request.

Alternatively, these methods need to be extended with useful argument types to make them easier to use. Users who want to use the lower-level Query interface can always generate a query to a given instance via the .query() factory and then join away as per the census module API.

Make object model docstrings Sphinx/Napoleon compatible

The object model docstrings are not using any cross-referencing yet, and they're also lacklustre regarding helpfulness to the user.

Conforming this documentation to the standard set for the other modules is a prerequisite to new users being able to use the object model effectively.

Rewrite discussion

I feel like picking up the threads where I left off some months ago: a full rewrite.

There are currently two parallel interfaces, one provides the low-level URL generation stuff (which should support any game with a DBG API), the other is the object-oriented, Pythonic model for PlanetSide 2.

I would like to start over with focussing on the Python end, making sure that is a nice, clean API without weird "query commands go into methods" decisions - the end user within Python shouldn't even have to think about query commands existing.

My instinct would be "add unit tests, fix the bugs, start rewriting and keep testing using the shiny new unit tests", but only if there is shared interest in making the API nicer to use.

[BUG] Methods and fields with the same iname in query.py and join.py

In auraxium/query.py, __init__() method create two arrays hide_fields and show_fields to store the fields given to the methods hide_fields() and show_fields().

Since the arrays and the methods have the same name, the arrays overwrite the methods, causing a TypeError: 'list' object is not callable when trying to call the methods.

The same happens in auraxium/join.py with both the arrays and the methods called hide and show.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.